Sep 17 17:40:55.721671 ip-10-0-129-186 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:40:59.156105 ip-10-0-129-186 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:59.156105 ip-10-0-129-186 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:40:59.156105 ip-10-0-129-186 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:59.156105 ip-10-0-129-186 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:40:59.156105 ip-10-0-129-186 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:59.156891 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.156194    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:40:59.160640 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160616    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:59.160640 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160636    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:59.160640 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160640    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:59.160640 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160643    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:59.160640 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160647    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160651    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160654    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160657    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160659    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160662    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160665    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160668    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160670    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160673    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160675    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160678    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160681    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160683    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160686    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160688    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160691    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160693    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160696    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160705    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:59.160832 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160708    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160710    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160713    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160715    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160718    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160722    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160725    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160728    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160732    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160737    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160740    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160743    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160746    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160749    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160753    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160757    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160760    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160762    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160765    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:59.161309 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160768    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160770    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160773    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160777    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160781    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160783    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160786    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160790    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160793    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160796    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160799    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160801    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160804    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160807    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160809    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160812    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160814    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160817    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160819    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:59.161784 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160822    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160825    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160827    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160830    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160833    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160836    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160839    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160841    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160846    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160849    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160852    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160854    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160857    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160860    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160862    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160865    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160868    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160871    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160873    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:59.162244 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160876    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160878    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160881    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160884    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160886    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160889    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160891    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160894    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160896    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160899    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160901    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160904    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160906    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160909    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160911    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160914    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160916    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.160918    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161626    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161633    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:59.162733 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161636    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161641    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161644    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161647    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161650    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161652    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161655    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161658    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161660    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161663    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161665    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161668    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161671    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161673    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161677    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161680    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161682    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161685    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161687    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161690    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:59.163207 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161693    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161698    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161700    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161704    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161708    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161712    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161715    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161718    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161721    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161724    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161727    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161730    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161732    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161736    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161739    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161741    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161744    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161747    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161750    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:59.163716 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161752    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161755    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161758    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161761    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161763    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161766    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161768    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161771    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161773    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161776    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161779    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161781    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161783    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161786    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161789    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161791    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161795    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161798    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161801    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161804    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:59.164181 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161806    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161809    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161811    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161814    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161817    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161819    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161822    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161825    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161827    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161830    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161833    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161835    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161838    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161840    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161843    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161846    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161848    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161851    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161854    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161856    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:59.164672 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161859    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161861    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161873    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161876    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161878    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161881    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161884    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161887    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161890    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161893    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161895    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161898    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161900    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161903    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161905    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161908    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161911    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.161914    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162299    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162305    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:59.165152 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162309    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162312    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162315    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162317    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162320    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162323    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162326    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162329    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162331    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162334    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162337    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162340    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162342    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162345    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162348    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162352    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162356    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162358    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162361    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:59.165670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162364    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162367    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162370    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162372    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162375    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162378    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162381    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162383    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162386    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162389    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162407    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162411    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162414    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162417    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162421    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162424    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162427    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162431    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162435    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:59.166139 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162438    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162441    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162444    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162446    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162449    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162451    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162454    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162457    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162467    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162471    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162473    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162476    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162479    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162482    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162484    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162487    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162489    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162492    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162495    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162497    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:59.166626 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162500    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162502    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162506    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162508    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162511    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162514    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162517    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162521    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162524    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162527    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162529    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162532    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162534    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162537    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162540    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162542    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162545    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162548    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162550    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162553    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162555    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:59.167144 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162558    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162561    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162564    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162567    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162570    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162572    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162575    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162578    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162580    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162583    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162586    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162588    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162591    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162593    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162596    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162598    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162601    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.162603    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162664    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162703    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162710    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:40:59.167668 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162714    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162719    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162722    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162726    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162731    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162734    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162737    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162741    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162744    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162748    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162751    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162753    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162756    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162763    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162766    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162771    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162774    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162777    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162780    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162784    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162787    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162790    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162793    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:40:59.168173 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162797    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162800    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162803    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162805    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162809    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162811    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162816    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162819    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162822    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162825    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162828    2565 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162831    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162835    2565 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162838    2565 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162841    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162844    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162846    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162850    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162853    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162856    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162859    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162862    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162864    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162868    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162871    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:40:59.168825 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162874    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162877    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162880    2565 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162884    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162886    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162889    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162892    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162895    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162898    2565 flags.go:64] FLAG: --help="false"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162901    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162904    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162907    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162910    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162913    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162917    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162920    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162923    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162926    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162929    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162932    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162935    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162938    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162941    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162944    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:40:59.169450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162947    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162950    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162952    2565 flags.go:64] FLAG: --lock-file=""
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162955    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162958    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162961    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162966    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162972    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162975    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162978    2565 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162981    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162985    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162988    2565 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162990    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162995    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.162998    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163002    2565 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163004    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163007    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163010    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163013    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163016    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163019    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163022    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163029    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:40:59.170037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163034    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163037    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163040    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163043    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163048    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163051    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163054    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163057    2565 flags.go:64] FLAG: --port="10250"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163060    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163063    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06043ca9c80b15f11"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163066    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163069    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163072    2565 flags.go:64] FLAG: --register-node="true"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163075    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163079    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163084    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163087    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163090    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163094    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163097    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163101    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163104    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163106    2565 flags.go:64] FLAG: --runonce="false"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163109    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163112    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:40:59.170662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163115    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163118    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163121    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163124    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163127    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163130    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163133    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163138    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163141    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163144    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163147    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163150    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163153    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163158    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163160    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163163    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163169    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163171    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163174    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163177    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163180    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163183    2565 flags.go:64] FLAG: --v="2"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163188    2565 flags.go:64] FLAG: --version="false"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163193    2565 flags.go:64] FLAG: --vmodule=""
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163197    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:40:59.171293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163200    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163317    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163321    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163324    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163327    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163330    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163333    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163336    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163339    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163344    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163347    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163350    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163353    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163356    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163360    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163362    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163365    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163368    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163370    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:59.171977 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163382    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163385    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163388    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163405    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163409    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163412    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163415    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163417    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163420    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163423    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163429    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163432    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163436    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163438    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163441    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163444    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163447    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163449    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163452    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163455    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:59.172489 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163458    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163460    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163463    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163467    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163470    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163473    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163476    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163480    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163483    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163485    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163488    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163491    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163494    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163496    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163499    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163502    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163504    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163507    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163510    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163512    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:59.172961 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163515    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163518    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163522    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163525    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163527    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163531    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163535    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163537    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163540    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163543    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163545    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163548    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163551    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163553    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163556    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163558    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163561    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163564    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163566    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163570    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:59.173452 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163573    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163575    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163578    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163581    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163583    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163586    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163588    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163591    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163594    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163596    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163599    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163601    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163604    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163607    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163611    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163615    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163618    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163621    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163625    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:59.173936 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163627    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.163631    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.163636    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.170245    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.170262    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.170282    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.170291    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170367    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170372    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170376    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170379    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170382    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170386    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170389    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170407    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:59.174389 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170411    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170415    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170419    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170422    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170425    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170428    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170431    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170433    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170436    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170439    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170442    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170444    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170447    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170449    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170452    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170455    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170457    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170466    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170469    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170472    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:59.174786 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170474    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170479    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170482    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170484    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170487    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170489    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170492    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170495    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170498    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170500    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170503    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170505    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170508    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170511    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170514    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170516    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170519    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170521    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170524    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170526    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:59.175267 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170529    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170531    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170534    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170536    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170539    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170542    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170544    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170547    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170549    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170552    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170555    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170559    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170562    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170566    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170569    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170572    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170575    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170578    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170580    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:59.175826 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170583    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170586    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170589    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170592    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170594    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170597    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170600    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170602    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170605    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170608    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170610    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170613    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170617    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170620    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170623    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170626    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170629    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170631    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170634    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170637    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:59.176298 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170639    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170642    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170645    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170647    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170650    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170653    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170656    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170659    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170662    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170665    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170669    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170671    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.170677    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170790    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170794    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:59.176862 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170797    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170800    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170802    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170805    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170808    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170810    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170813    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170816    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170818    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170821    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170823    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170826    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170828    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170831    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170834    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170836    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170839    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170841    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170843    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170846    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:59.177242 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170848    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170851    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170853    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170856    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170860    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170862    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170865    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170868    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170870    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170873    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170875    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170878    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170881    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170885    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170888    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170892    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170896    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170899    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170902    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:59.177763 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170904    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170907    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170910    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170912    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170915    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170917    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170920    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170922    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170925    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170927    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170930    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170932    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170935    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170937    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170940    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170943    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170946    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170948    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170951    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170953    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:59.178571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170956    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170959    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170962    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170965    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170968    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170971    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170974    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170976    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170979    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170981    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170984    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170986    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170989    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170991    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170994    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170996    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.170999    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171001    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171004    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:59.179237 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171006    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171009    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171012    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171014    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171017    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171019    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171022    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171024    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171027    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171030    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171033    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171035    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171038    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171040    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171043    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171046    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171048    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171051    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:59.179867 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:40:59.171053    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:59.180304 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.171059    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:59.180304 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.171153    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:40:59.180304 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.173011    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:40:59.180304 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.173753    2565 server.go:1019] "Starting client certificate rotation"
Sep 17 17:40:59.180304 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.173850    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:40:59.180304 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.173882    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:40:59.180304 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.175972    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:40:59.180304 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.177671    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:40:59.184638 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.184621    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:40:59.186804 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.186787    2565 log.go:25] "Validated CRI v1 image API"
Sep 17 17:40:59.187680 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.187664    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:40:59.189884 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.189868    2565 fs.go:135] Filesystem UUIDs: map[372e9d84-ed5b-433f-9746-acd63c00c5ae:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7a51eb71-3ff3-4c09-9606-00940f306d02:/dev/nvme0n1p4]
Sep 17 17:40:59.189969 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.189883    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:40:59.194606 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.194501    2565 manager.go:216] Machine: {Timestamp:2026-09-17 17:40:59.193424375 +0000 UTC m=+2.029193589 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100163 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25a57659d06af4a4b6ed89b04613f2 SystemUUID:ec25a576-59d0-6af4-a4b6-ed89b04613f2 BootID:fea9f377-9e9d-477b-a7bb-f80ff9a5aed7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:20:56:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:20:56:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:c3:57:ae:83:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:40:59.194606 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.194603    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:40:59.194712 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.194703    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:40:59.196089 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.196069    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:40:59.196247 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.196090    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-186.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:40:59.196734 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.196724    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:40:59.196766 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.196737    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:40:59.196766 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.196744    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:40:59.196766 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.196755    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:40:59.196847 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.196822    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:40:59.199773 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.199761    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:40:59.199899 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.199889    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:40:59.199935 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.199924    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:40:59.199962 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.199938    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:40:59.199962 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.199951    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:40:59.199962 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.199959    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:40:59.200065 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.199968    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:40:59.200294 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.200274    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:40:59.200338 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.200301    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:40:59.202191 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.202174    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:40:59.204146 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.204131    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:40:59.204238 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.204169    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:40:59.205189 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205176    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:40:59.205267 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205195    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:40:59.205267 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205204    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:40:59.205267 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205212    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:40:59.205267 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205224    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:40:59.205267 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205232    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:40:59.205267 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205251    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:40:59.205267 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205259    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:40:59.205529 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205276    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:40:59.205529 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205285    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:40:59.205529 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205304    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:40:59.205529 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205313    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:40:59.205529 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205344    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:40:59.205529 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.205353    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:40:59.208841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.208827    2565 server.go:1268] "Started kubelet"
Sep 17 17:40:59.208935 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.208898    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:40:59.208980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.208926    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:40:59.209018 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.208995    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:40:59.209625 ip-10-0-129-186 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:40:59.210553 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.210533    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:40:59.210641 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.210602    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:40:59.242000 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.241976    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:40:59.243138 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.243119    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-186.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:40:59.243932 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.243909    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:40:59.243997 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.243909    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:40:59.244501 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.244426    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:40:59.244586 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.244530    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:40:59.244668 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.244648    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:40:59.244738 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.244684    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:40:59.244738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.244702    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-th2sj"
Sep 17 17:40:59.245228 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245204    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:40:59.245341 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245247    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:40:59.245341 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245249    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:40:59.245341 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245337    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:40:59.245511 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245344    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:40:59.245575 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245560    2565 factory.go:55] Registering systemd factory
Sep 17 17:40:59.245635 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245578    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:40:59.245836 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245816    2565 factory.go:153] Registering CRI-O factory
Sep 17 17:40:59.245899 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245835    2565 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:40:59.245899 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245892    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:40:59.245995 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245931    2565 factory.go:103] Registering Raw factory
Sep 17 17:40:59.245995 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.245947    2565 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:40:59.246780 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.246761    2565 manager.go:349] Starting recovery of all containers
Sep 17 17:40:59.248032 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.248009    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:40:59.248930 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.247978    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-186.ec2.internal.18d62c86c16e073e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-186.ec2.internal,UID:ip-10-0-129-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-186.ec2.internal,},FirstTimestamp:2026-09-17 17:40:59.208804158 +0000 UTC m=+2.044573372,LastTimestamp:2026-09-17 17:40:59.208804158 +0000 UTC m=+2.044573372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-186.ec2.internal,}"
Sep 17 17:40:59.249330 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.249298    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:40:59.251557 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.251528    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-th2sj"
Sep 17 17:40:59.256305 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.256292    2565 manager.go:354] Recovery completed
Sep 17 17:40:59.260446 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.260432    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:59.262565 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.262551    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:59.262640 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.262582    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:59.262640 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.262594    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:59.263127 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.263115    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:40:59.263127 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.263126    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:40:59.263227 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.263140    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:40:59.263227 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.263189    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:59.263942 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.263926    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:59.264025 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.263957    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:59.264025 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.263972    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:59.264153 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.264082    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-186.ec2.internal.18d62c86c4a26c7b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-186.ec2.internal,UID:ip-10-0-129-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-186.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-186.ec2.internal,},FirstTimestamp:2026-09-17 17:40:59.262569595 +0000 UTC m=+2.098338809,LastTimestamp:2026-09-17 17:40:59.262569595 +0000 UTC m=+2.098338809,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-186.ec2.internal,}"
Sep 17 17:40:59.264420 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.264407    2565 policy_none.go:49] "None policy: Start"
Sep 17 17:40:59.264459 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.264426    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:40:59.264459 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.264437    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:40:59.265432 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.265421    2565 policy_none.go:47] "Start"
Sep 17 17:40:59.308922 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.308900    2565 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.308936    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.308946    2565 server.go:87] "Starting device plugin registration server"
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.309198    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.309215    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.309303    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.309370    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.309378    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.309825    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:40:59.313582 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.309871    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:40:59.341625 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.341595    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:40:59.342957 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.342935    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:40:59.343040 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.342980    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:40:59.343040 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.343000    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:40:59.343109 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.343047    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:40:59.345522 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.345503    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:59.409807 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.409763    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:59.410701 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.410686    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:59.410781 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.410718    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:59.410781 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.410732    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:59.410781 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.410763    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.418922 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.418908    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.418989 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.418927    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-186.ec2.internal\": node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:40:59.438262 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.438241    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:40:59.443325 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.443301    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"]
Sep 17 17:40:59.443374 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.443365    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:59.444137 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.444124    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:59.444193 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.444149    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:59.444193 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.444158    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:59.445441 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.445430    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:59.445568 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.445544    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.445625 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.445584    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:59.445625 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.445545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/15411fab6b97a438488300fab1eaf100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"15411fab6b97a438488300fab1eaf100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.445625 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.445619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15411fab6b97a438488300fab1eaf100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"15411fab6b97a438488300fab1eaf100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.445760 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.445635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81daa90f8e3c12260694003f97da072d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-186.ec2.internal\" (UID: \"81daa90f8e3c12260694003f97da072d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.446111 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.446098    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:59.446181 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.446125    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:59.446181 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.446140    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:59.446256 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.446127    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:59.446256 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.446224    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:59.446256 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.446235    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:59.447334 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.447319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.447404 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.447351    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:59.447988 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.447969    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:59.448040 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.447991    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:59.448040 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.448004    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:59.474991 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.474969    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-186.ec2.internal\" not found" node="ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.479206 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.479191    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-186.ec2.internal\" not found" node="ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.538508 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.538489    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:40:59.546624 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.546607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/15411fab6b97a438488300fab1eaf100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"15411fab6b97a438488300fab1eaf100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.546738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.546629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15411fab6b97a438488300fab1eaf100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"15411fab6b97a438488300fab1eaf100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.546738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.546645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81daa90f8e3c12260694003f97da072d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-186.ec2.internal\" (UID: \"81daa90f8e3c12260694003f97da072d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.546738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.546706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81daa90f8e3c12260694003f97da072d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-186.ec2.internal\" (UID: \"81daa90f8e3c12260694003f97da072d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.546834 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.546734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15411fab6b97a438488300fab1eaf100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"15411fab6b97a438488300fab1eaf100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.546834 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.546707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/15411fab6b97a438488300fab1eaf100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"15411fab6b97a438488300fab1eaf100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.639176 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.639150    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:40:59.739962 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.739882    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:40:59.778439 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.778381    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.783992 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:40:59.783977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Sep 17 17:40:59.841047 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.841015    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:40:59.941596 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:40:59.941564    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:41:00.042153 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:00.042066    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:41:00.142747 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:00.142716    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:41:00.174165 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.174141    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:41:00.174575 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.174270    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:41:00.243865 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:00.243675    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:41:00.255637 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.255608    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:35:59 +0000 UTC" deadline="2028-03-12 15:26:18.802755328 +0000 UTC"
Sep 17 17:41:00.255637 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.255634    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13005h45m18.547124089s"
Sep 17 17:41:00.343972 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:00.343940    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:41:00.376651 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:00.376619    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81daa90f8e3c12260694003f97da072d.slice/crio-4b17c2e6d912d3a0d0da7564d0bb23365fb1fc56ede285b394e11951e86893c1 WatchSource:0}: Error finding container 4b17c2e6d912d3a0d0da7564d0bb23365fb1fc56ede285b394e11951e86893c1: Status 404 returned error can't find the container with id 4b17c2e6d912d3a0d0da7564d0bb23365fb1fc56ede285b394e11951e86893c1
Sep 17 17:41:00.377212 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:00.377186    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15411fab6b97a438488300fab1eaf100.slice/crio-c97d0d68e4005e850dc51a18abac9cd2b372ae2883e7ce202738721d7431d7d1 WatchSource:0}: Error finding container c97d0d68e4005e850dc51a18abac9cd2b372ae2883e7ce202738721d7431d7d1: Status 404 returned error can't find the container with id c97d0d68e4005e850dc51a18abac9cd2b372ae2883e7ce202738721d7431d7d1
Sep 17 17:41:00.381525 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.381502    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:00.444600 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:00.444575    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:41:00.545068 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:00.545042    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:41:00.585476 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.585457    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:00.645405 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:00.645332    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Sep 17 17:41:00.713658 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.713622    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:00.727144 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.727120    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:00.744883 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.744847    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Sep 17 17:41:00.759800 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.759783    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:00.760732 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.760720    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Sep 17 17:41:00.770582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:00.770562    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:01.201390 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.201350    2565 apiserver.go:52] "Watching apiserver"
Sep 17 17:41:01.207214 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.207024    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:41:01.207424 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.207387    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-lpzgv","openshift-multus/network-metrics-daemon-x9dql","openshift-network-diagnostics/network-check-target-5gtth","openshift-network-operator/iptables-alerter-wmcq4","kube-system/konnectivity-agent-s7zjm","openshift-cluster-node-tuning-operator/tuned-kbdfg","openshift-dns/node-resolver-bvd5m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal","openshift-multus/multus-additional-cni-plugins-zvpl4","openshift-ovn-kubernetes/ovnkube-node-jk425","kube-system/global-pull-secret-syncer-v47zx","kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz","openshift-image-registry/node-ca-2vkjk"]
Sep 17 17:41:01.209145 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.209117    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.210570 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.210546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.211351 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.211331    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:41:01.211456 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.211334    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:41:01.211456 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.211338    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kb52x\""
Sep 17 17:41:01.213518 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.213498    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:01.213611 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.213592    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:41:01.213731 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.213609    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:01.213798 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.213731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:41:01.214156 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.213976    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b6jq5\""
Sep 17 17:41:01.214156 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.213999    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:41:01.214301 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.214263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.215632 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.215612    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:01.216850 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.216832    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z6cb9\""
Sep 17 17:41:01.217322 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.217040    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:01.217322 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.217184    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:01.217628 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.217446    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.218972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.218953    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.219943 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.219917    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:41:01.220702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.220185    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:41:01.220702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.220267    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:01.220702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.220384    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:41:01.220702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.220491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-drqnh\""
Sep 17 17:41:01.220702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.220575    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cbmx2\""
Sep 17 17:41:01.220702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.220657    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:01.220999 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.220839    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.221644 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.221495    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:41:01.221734 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.221672    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-llgzl\""
Sep 17 17:41:01.222980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.222017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:41:01.222980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.222053    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:41:01.222980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.222743    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:41:01.222980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.222781    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jw6qb\""
Sep 17 17:41:01.222980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.222795    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:41:01.222980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.222811    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:01.222980 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.222868    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:01.222980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.222961    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:41:01.224145 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.224126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.226087 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.226068    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.226180 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.226145    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:01.226240 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.226220    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:41:01.226293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.226273    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ktglk\""
Sep 17 17:41:01.226389 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.226373    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:41:01.228746 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.227053    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:41:01.228746 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.227056    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:41:01.228746 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.227297    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:41:01.228746 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.228139    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.228746 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.228179    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:41:01.230939 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.230889    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:41:01.230939 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.230922    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cptkl\""
Sep 17 17:41:01.231127 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.231110    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:41:01.231186 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.231116    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:41:01.245686 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.245666    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:01.246503 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.246484    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:41:01.255418 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:01.255513 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ce68719-e9b2-4fee-b1d2-562bd57bca45-hosts-file\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.255513 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-cni-bin\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.255513 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255500    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:01.255667 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94f278e6-8be6-42ba-b87a-be298b200738-multus-daemon-config\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.255667 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-serviceca\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.255667 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/172b0396-6b46-453f-b279-c37f1b0cc403-iptables-alerter-script\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.255667 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-system-cni-dir\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.255667 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-kubelet\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-env-overrides\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255722    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtnqr\" (UniqueName: \"kubernetes.io/projected/172b0396-6b46-453f-b279-c37f1b0cc403-kube-api-access-dtnqr\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255748    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-systemd\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-host\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzjk\" (UniqueName: \"kubernetes.io/projected/06e9329e-dfb9-4ca8-9807-995754f7f505-kube-api-access-dhzjk\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-cni-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-sys-fs\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.255901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-modprobe-d\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.255987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-run-ovn-kubernetes\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovnkube-config\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m595q\" (UniqueName: \"kubernetes.io/projected/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-kube-api-access-m595q\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-device-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysctl-d\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-log-socket\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-system-cni-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxlj\" (UniqueName: \"kubernetes.io/projected/94f278e6-8be6-42ba-b87a-be298b200738-kube-api-access-dxxlj\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.256306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/172b0396-6b46-453f-b279-c37f1b0cc403-host-slash\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0868d5ee-d875-45d7-866e-5678b9f1411d-konnectivity-ca\") pod \"konnectivity-agent-s7zjm\" (UID: \"0868d5ee-d875-45d7-866e-5678b9f1411d\") " pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256375    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-run\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-lib-modules\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hn4\" (UniqueName: \"kubernetes.io/projected/c3415d80-dec7-47e2-aded-58c3bd05537f-kube-api-access-z9hn4\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-var-lib-kubelet\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3415e108-2762-4d68-9cdc-10f826382faf-etc-tuned\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-etc-kubernetes\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45898ed0-e311-492a-ab02-7851051c85a9-kubelet-config\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.256738 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovn-node-metrics-cert\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94f278e6-8be6-42ba-b87a-be298b200738-cni-binary-copy\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-k8s-cni-cncf-io\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfk4d\" (UniqueName: \"kubernetes.io/projected/1ce68719-e9b2-4fee-b1d2-562bd57bca45-kube-api-access-nfk4d\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256906    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-socket-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-sys\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.256987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksd4x\" (UniqueName: \"kubernetes.io/projected/3415e108-2762-4d68-9cdc-10f826382faf-kube-api-access-ksd4x\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-node-log\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-cni-netd\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257091    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-host\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-systemd-units\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovnkube-script-lib\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-netns\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-hostroot\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnlb\" (UniqueName: \"kubernetes.io/projected/16cd028c-cfdd-4f39-8fd2-647ddad31d62-kube-api-access-6lnlb\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.257273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0868d5ee-d875-45d7-866e-5678b9f1411d-agent-certs\") pod \"konnectivity-agent-s7zjm\" (UID: \"0868d5ee-d875-45d7-866e-5678b9f1411d\") " pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysconfig\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-systemd\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-var-lib-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce68719-e9b2-4fee-b1d2-562bd57bca45-tmp-dir\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-multus-certs\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45898ed0-e311-492a-ab02-7851051c85a9-dbus\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-kubernetes\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-cnibin\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-etc-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-cnibin\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-socket-dir-parent\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-cni-multus\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-run-netns\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-ovn\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.257984 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs8q\" (UniqueName: \"kubernetes.io/projected/ef7c1c2f-a805-434a-9537-bed7df1e3570-kube-api-access-nqs8q\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-slash\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257805    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-cni-bin\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-kubelet\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-conf-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-registration-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysctl-conf\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3415e108-2762-4d68-9cdc-10f826382faf-tmp\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.258665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.257952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.300239 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.300219    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jlrtp"
Sep 17 17:41:01.310479 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.310451    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jlrtp"
Sep 17 17:41:01.348525 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.348471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal" event={"ID":"15411fab6b97a438488300fab1eaf100","Type":"ContainerStarted","Data":"c97d0d68e4005e850dc51a18abac9cd2b372ae2883e7ce202738721d7431d7d1"}
Sep 17 17:41:01.349853 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.349831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal" event={"ID":"81daa90f8e3c12260694003f97da072d","Type":"ContainerStarted","Data":"4b17c2e6d912d3a0d0da7564d0bb23365fb1fc56ede285b394e11951e86893c1"}
Sep 17 17:41:01.358535 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.358494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0868d5ee-d875-45d7-866e-5678b9f1411d-agent-certs\") pod \"konnectivity-agent-s7zjm\" (UID: \"0868d5ee-d875-45d7-866e-5678b9f1411d\") " pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:01.358535 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.358530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysconfig\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.359378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-systemd\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.359624    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-systemd\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.359731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysconfig\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.359794    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.359835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-var-lib-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.359934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-var-lib-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce68719-e9b2-4fee-b1d2-562bd57bca45-tmp-dir\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-multus-certs\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45898ed0-e311-492a-ab02-7851051c85a9-dbus\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.360168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-kubernetes\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-cnibin\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-etc-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-cnibin\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-socket-dir-parent\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-cni-multus\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-run-netns\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-ovn\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs8q\" (UniqueName: \"kubernetes.io/projected/ef7c1c2f-a805-434a-9537-bed7df1e3570-kube-api-access-nqs8q\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-etc-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-socket-dir-parent\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.360702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45898ed0-e311-492a-ab02-7851051c85a9-dbus\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-cni-multus\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-kubernetes\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.360880    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-cnibin\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.360983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-slash\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.360998    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret podName:45898ed0-e311-492a-ab02-7851051c85a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:01.860969235 +0000 UTC m=+4.696738439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret") pod "global-pull-secret-syncer-v47zx" (UID: "45898ed0-e311-492a-ab02-7851051c85a9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361017    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-slash\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce68719-e9b2-4fee-b1d2-562bd57bca45-tmp-dir\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-cni-bin\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-multus-certs\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-kubelet\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-conf-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-run-netns\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-registration-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-ovn\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.361237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysctl-conf\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-kubelet\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-conf-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-registration-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3415e108-2762-4d68-9cdc-10f826382faf-tmp\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-cni-bin\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ce68719-e9b2-4fee-b1d2-562bd57bca45-hosts-file\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-cnibin\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-cni-bin\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ce68719-e9b2-4fee-b1d2-562bd57bca45-hosts-file\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94f278e6-8be6-42ba-b87a-be298b200738-multus-daemon-config\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-serviceca\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/172b0396-6b46-453f-b279-c37f1b0cc403-iptables-alerter-script\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-system-cni-dir\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-kubelet\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.362091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-env-overrides\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.361988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtnqr\" (UniqueName: \"kubernetes.io/projected/172b0396-6b46-453f-b279-c37f1b0cc403-kube-api-access-dtnqr\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-systemd\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-host\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzjk\" (UniqueName: \"kubernetes.io/projected/06e9329e-dfb9-4ca8-9807-995754f7f505-kube-api-access-dhzjk\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-cni-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-sys-fs\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362186    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-modprobe-d\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-run-ovn-kubernetes\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovnkube-config\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m595q\" (UniqueName: \"kubernetes.io/projected/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-kube-api-access-m595q\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-device-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-host\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.362889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysctl-d\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-modprobe-d\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-log-socket\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-device-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362808    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysctl-d\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-run-openvswitch\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.362981    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-multus-cni-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363043    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-sys-fs\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-run-ovn-kubernetes\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.363494    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.363549    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs podName:06e9329e-dfb9-4ca8-9807-995754f7f505 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:01.863525491 +0000 UTC m=+4.699294699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs") pod "network-metrics-daemon-x9dql" (UID: "06e9329e-dfb9-4ca8-9807-995754f7f505") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/172b0396-6b46-453f-b279-c37f1b0cc403-iptables-alerter-script\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363663    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-sysctl-conf\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-system-cni-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxlj\" (UniqueName: \"kubernetes.io/projected/94f278e6-8be6-42ba-b87a-be298b200738-kube-api-access-dxxlj\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/172b0396-6b46-453f-b279-c37f1b0cc403-host-slash\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.363816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0868d5ee-d875-45d7-866e-5678b9f1411d-konnectivity-ca\") pod \"konnectivity-agent-s7zjm\" (UID: \"0868d5ee-d875-45d7-866e-5678b9f1411d\") " pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-run\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-lib-modules\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363929    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovnkube-config\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hn4\" (UniqueName: \"kubernetes.io/projected/c3415d80-dec7-47e2-aded-58c3bd05537f-kube-api-access-z9hn4\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.363988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-system-cni-dir\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/172b0396-6b46-453f-b279-c37f1b0cc403-host-slash\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-log-socket\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-run\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.364605 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364531    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-kubelet\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364635    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-lib-modules\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-etc-systemd\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-var-lib-kubelet\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94f278e6-8be6-42ba-b87a-be298b200738-multus-daemon-config\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3415e108-2762-4d68-9cdc-10f826382faf-etc-tuned\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-serviceca\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-etc-kubernetes\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-etc-kubernetes\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3415d80-dec7-47e2-aded-58c3bd05537f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.364988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45898ed0-e311-492a-ab02-7851051c85a9-kubelet-config\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365010    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.365161 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365082    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-var-lib-kubelet\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-env-overrides\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovn-node-metrics-cert\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94f278e6-8be6-42ba-b87a-be298b200738-cni-binary-copy\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-k8s-cni-cncf-io\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfk4d\" (UniqueName: \"kubernetes.io/projected/1ce68719-e9b2-4fee-b1d2-562bd57bca45-kube-api-access-nfk4d\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-socket-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365686    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-sys\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksd4x\" (UniqueName: \"kubernetes.io/projected/3415e108-2762-4d68-9cdc-10f826382faf-kube-api-access-ksd4x\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.365775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-node-log\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3415d80-dec7-47e2-aded-58c3bd05537f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-cni-netd\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-var-lib-cni-bin\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365940    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16cd028c-cfdd-4f39-8fd2-647ddad31d62-socket-dir\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365954    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-system-cni-dir\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.365976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-cni-netd\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3415e108-2762-4d68-9cdc-10f826382faf-sys\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-node-log\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-host\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-systemd-units\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.366289 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-k8s-cni-cncf-io\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.366918 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3415e108-2762-4d68-9cdc-10f826382faf-tmp\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.366918 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.366918 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45898ed0-e311-492a-ab02-7851051c85a9-kubelet-config\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.366918 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366633    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-host\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.366918 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94f278e6-8be6-42ba-b87a-be298b200738-cni-binary-copy\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.366918 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.366678    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ef7c1c2f-a805-434a-9537-bed7df1e3570-systemd-units\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.367374 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.367352    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3415e108-2762-4d68-9cdc-10f826382faf-etc-tuned\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.367968 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.367714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovnkube-script-lib\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.367968 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.367908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0868d5ee-d875-45d7-866e-5678b9f1411d-konnectivity-ca\") pod \"konnectivity-agent-s7zjm\" (UID: \"0868d5ee-d875-45d7-866e-5678b9f1411d\") " pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:01.368163 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.368143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovnkube-script-lib\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.368251 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.368192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-netns\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.368251 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.368232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-hostroot\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.368344 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.368262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnlb\" (UniqueName: \"kubernetes.io/projected/16cd028c-cfdd-4f39-8fd2-647ddad31d62-kube-api-access-6lnlb\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.368522 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.368470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-host-run-netns\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.369530 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.368534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94f278e6-8be6-42ba-b87a-be298b200738-hostroot\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.370174 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.369822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0868d5ee-d875-45d7-866e-5678b9f1411d-agent-certs\") pod \"konnectivity-agent-s7zjm\" (UID: \"0868d5ee-d875-45d7-866e-5678b9f1411d\") " pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:01.370174 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.370091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef7c1c2f-a805-434a-9537-bed7df1e3570-ovn-node-metrics-cert\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.371687 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.371664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs8q\" (UniqueName: \"kubernetes.io/projected/ef7c1c2f-a805-434a-9537-bed7df1e3570-kube-api-access-nqs8q\") pod \"ovnkube-node-jk425\" (UID: \"ef7c1c2f-a805-434a-9537-bed7df1e3570\") " pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.372852 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.372830    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:01.372977 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.372858    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:01.372977 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.372873    2565 projected.go:196] Error preparing data for projected volume kube-api-access-27rpg for pod openshift-network-diagnostics/network-check-target-5gtth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:01.372977 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.372947    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg podName:d1b48b76-797b-4760-9783-3a1b06ea75b7 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:01.872927333 +0000 UTC m=+4.708696553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-27rpg" (UniqueName: "kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg") pod "network-check-target-5gtth" (UID: "d1b48b76-797b-4760-9783-3a1b06ea75b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:01.373744 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.373725    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m595q\" (UniqueName: \"kubernetes.io/projected/fab829e2-cb5a-4779-bff8-0a6ae8a78c3f-kube-api-access-m595q\") pod \"node-ca-2vkjk\" (UID: \"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f\") " pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.374478 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.374426    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtnqr\" (UniqueName: \"kubernetes.io/projected/172b0396-6b46-453f-b279-c37f1b0cc403-kube-api-access-dtnqr\") pod \"iptables-alerter-wmcq4\" (UID: \"172b0396-6b46-453f-b279-c37f1b0cc403\") " pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.375631 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.375609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzjk\" (UniqueName: \"kubernetes.io/projected/06e9329e-dfb9-4ca8-9807-995754f7f505-kube-api-access-dhzjk\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:01.380236 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.380192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksd4x\" (UniqueName: \"kubernetes.io/projected/3415e108-2762-4d68-9cdc-10f826382faf-kube-api-access-ksd4x\") pod \"tuned-kbdfg\" (UID: \"3415e108-2762-4d68-9cdc-10f826382faf\") " pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.380593 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.380573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfk4d\" (UniqueName: \"kubernetes.io/projected/1ce68719-e9b2-4fee-b1d2-562bd57bca45-kube-api-access-nfk4d\") pod \"node-resolver-bvd5m\" (UID: \"1ce68719-e9b2-4fee-b1d2-562bd57bca45\") " pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.382597 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.382579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hn4\" (UniqueName: \"kubernetes.io/projected/c3415d80-dec7-47e2-aded-58c3bd05537f-kube-api-access-z9hn4\") pod \"multus-additional-cni-plugins-zvpl4\" (UID: \"c3415d80-dec7-47e2-aded-58c3bd05537f\") " pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.386240 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.386219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxlj\" (UniqueName: \"kubernetes.io/projected/94f278e6-8be6-42ba-b87a-be298b200738-kube-api-access-dxxlj\") pod \"multus-lpzgv\" (UID: \"94f278e6-8be6-42ba-b87a-be298b200738\") " pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.387708 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.387671    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnlb\" (UniqueName: \"kubernetes.io/projected/16cd028c-cfdd-4f39-8fd2-647ddad31d62-kube-api-access-6lnlb\") pod \"aws-ebs-csi-driver-node-4n5gz\" (UID: \"16cd028c-cfdd-4f39-8fd2-647ddad31d62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.522803 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.522725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bvd5m"
Sep 17 17:41:01.541575 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.541547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz"
Sep 17 17:41:01.549424 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.549403    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wmcq4"
Sep 17 17:41:01.556003 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.555987    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:01.563676 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.563659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kbdfg"
Sep 17 17:41:01.568296 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.568279    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:01.571444 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.571421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvpl4"
Sep 17 17:41:01.579782 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.579749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lpzgv"
Sep 17 17:41:01.586959 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.586941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:01.592167 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.592152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2vkjk"
Sep 17 17:41:01.871837 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.871756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:01.871837 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.871804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:01.872024 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.871895    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:01.872024 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.871907    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:01.872024 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.871946    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs podName:06e9329e-dfb9-4ca8-9807-995754f7f505 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:02.871934173 +0000 UTC m=+5.707703375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs") pod "network-metrics-daemon-x9dql" (UID: "06e9329e-dfb9-4ca8-9807-995754f7f505") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:01.872024 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.871964    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret podName:45898ed0-e311-492a-ab02-7851051c85a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:02.871955385 +0000 UTC m=+5.707724586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret") pod "global-pull-secret-syncer-v47zx" (UID: "45898ed0-e311-492a-ab02-7851051c85a9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:01.972870 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:01.972834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:01.973022 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.972992    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:01.973022 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.973011    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:01.973022 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.973020    2565 projected.go:196] Error preparing data for projected volume kube-api-access-27rpg for pod openshift-network-diagnostics/network-check-target-5gtth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:01.973154 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:01.973063    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg podName:d1b48b76-797b-4760-9783-3a1b06ea75b7 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:02.973051361 +0000 UTC m=+5.808820567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-27rpg" (UniqueName: "kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg") pod "network-check-target-5gtth" (UID: "d1b48b76-797b-4760-9783-3a1b06ea75b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:02.193576 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.193553    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cd028c_cfdd_4f39_8fd2_647ddad31d62.slice/crio-cca756b3a474d384e1f3d80eec732eeae395ea67fdd7acd5773720126653e5e5 WatchSource:0}: Error finding container cca756b3a474d384e1f3d80eec732eeae395ea67fdd7acd5773720126653e5e5: Status 404 returned error can't find the container with id cca756b3a474d384e1f3d80eec732eeae395ea67fdd7acd5773720126653e5e5
Sep 17 17:41:02.194408 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.194346    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0868d5ee_d875_45d7_866e_5678b9f1411d.slice/crio-c944cf560eafa558073644c78d66b3f30f1475840f48d3e75bce7ef58f148a64 WatchSource:0}: Error finding container c944cf560eafa558073644c78d66b3f30f1475840f48d3e75bce7ef58f148a64: Status 404 returned error can't find the container with id c944cf560eafa558073644c78d66b3f30f1475840f48d3e75bce7ef58f148a64
Sep 17 17:41:02.195414 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.195308    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3415e108_2762_4d68_9cdc_10f826382faf.slice/crio-2a10cc02d3a60e57ceacb731048817db62738078f415dd4d15ae54029aa36367 WatchSource:0}: Error finding container 2a10cc02d3a60e57ceacb731048817db62738078f415dd4d15ae54029aa36367: Status 404 returned error can't find the container with id 2a10cc02d3a60e57ceacb731048817db62738078f415dd4d15ae54029aa36367
Sep 17 17:41:02.197065 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.197040    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3415d80_dec7_47e2_aded_58c3bd05537f.slice/crio-09108d49474b1b052db908a7d1b110cda9fca933bf8c4c00bafc2035ff3752ad WatchSource:0}: Error finding container 09108d49474b1b052db908a7d1b110cda9fca933bf8c4c00bafc2035ff3752ad: Status 404 returned error can't find the container with id 09108d49474b1b052db908a7d1b110cda9fca933bf8c4c00bafc2035ff3752ad
Sep 17 17:41:02.198815 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.198689    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172b0396_6b46_453f_b279_c37f1b0cc403.slice/crio-acba9958596f5d8b9c65a30536113b44c5d73b83d723fa208fbcd943475edd82 WatchSource:0}: Error finding container acba9958596f5d8b9c65a30536113b44c5d73b83d723fa208fbcd943475edd82: Status 404 returned error can't find the container with id acba9958596f5d8b9c65a30536113b44c5d73b83d723fa208fbcd943475edd82
Sep 17 17:41:02.201972 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.201948    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab829e2_cb5a_4779_bff8_0a6ae8a78c3f.slice/crio-1827be6762929416e755bdf9539af754fbbe68070deba834e87303c9caec3fcb WatchSource:0}: Error finding container 1827be6762929416e755bdf9539af754fbbe68070deba834e87303c9caec3fcb: Status 404 returned error can't find the container with id 1827be6762929416e755bdf9539af754fbbe68070deba834e87303c9caec3fcb
Sep 17 17:41:02.202842 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.202821    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f278e6_8be6_42ba_b87a_be298b200738.slice/crio-24cd6ec761eb4ab50b1e456e134ca4857deee9bf65454586c9c287a17362e795 WatchSource:0}: Error finding container 24cd6ec761eb4ab50b1e456e134ca4857deee9bf65454586c9c287a17362e795: Status 404 returned error can't find the container with id 24cd6ec761eb4ab50b1e456e134ca4857deee9bf65454586c9c287a17362e795
Sep 17 17:41:02.203600 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.203578    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7c1c2f_a805_434a_9537_bed7df1e3570.slice/crio-fea5f9abf9271b8b323ebe683ed41b4d7d5b0d8ce35804b3534d58d1414ce48a WatchSource:0}: Error finding container fea5f9abf9271b8b323ebe683ed41b4d7d5b0d8ce35804b3534d58d1414ce48a: Status 404 returned error can't find the container with id fea5f9abf9271b8b323ebe683ed41b4d7d5b0d8ce35804b3534d58d1414ce48a
Sep 17 17:41:02.205348 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:02.205060    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce68719_e9b2_4fee_b1d2_562bd57bca45.slice/crio-aa2120b9846d34e8eb4c4445e20dfc040399fe0659d39aa6360e16417dabdd06 WatchSource:0}: Error finding container aa2120b9846d34e8eb4c4445e20dfc040399fe0659d39aa6360e16417dabdd06: Status 404 returned error can't find the container with id aa2120b9846d34e8eb4c4445e20dfc040399fe0659d39aa6360e16417dabdd06
Sep 17 17:41:02.312366 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.312332    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:01 +0000 UTC" deadline="2028-06-25 07:59:58.9390238 +0000 UTC"
Sep 17 17:41:02.312366 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.312367    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15518h18m56.626661687s"
Sep 17 17:41:02.344057 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.344031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:02.344178 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.344154    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:02.344256 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.344201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:02.344334 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.344316    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:02.353382 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.353359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal" event={"ID":"81daa90f8e3c12260694003f97da072d","Type":"ContainerStarted","Data":"b1beefba34b675dd8dfbc980d4abcba7be41a1c60dc38e16a51ee87e2ff3d535"}
Sep 17 17:41:02.354428 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.354406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bvd5m" event={"ID":"1ce68719-e9b2-4fee-b1d2-562bd57bca45","Type":"ContainerStarted","Data":"aa2120b9846d34e8eb4c4445e20dfc040399fe0659d39aa6360e16417dabdd06"}
Sep 17 17:41:02.355635 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.355609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"fea5f9abf9271b8b323ebe683ed41b4d7d5b0d8ce35804b3534d58d1414ce48a"}
Sep 17 17:41:02.356777 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.356753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kbdfg" event={"ID":"3415e108-2762-4d68-9cdc-10f826382faf","Type":"ContainerStarted","Data":"2a10cc02d3a60e57ceacb731048817db62738078f415dd4d15ae54029aa36367"}
Sep 17 17:41:02.358056 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.358022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz" event={"ID":"16cd028c-cfdd-4f39-8fd2-647ddad31d62","Type":"ContainerStarted","Data":"cca756b3a474d384e1f3d80eec732eeae395ea67fdd7acd5773720126653e5e5"}
Sep 17 17:41:02.359127 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.359109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lpzgv" event={"ID":"94f278e6-8be6-42ba-b87a-be298b200738","Type":"ContainerStarted","Data":"24cd6ec761eb4ab50b1e456e134ca4857deee9bf65454586c9c287a17362e795"}
Sep 17 17:41:02.360104 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.360083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2vkjk" event={"ID":"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f","Type":"ContainerStarted","Data":"1827be6762929416e755bdf9539af754fbbe68070deba834e87303c9caec3fcb"}
Sep 17 17:41:02.361181 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.361161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wmcq4" event={"ID":"172b0396-6b46-453f-b279-c37f1b0cc403","Type":"ContainerStarted","Data":"acba9958596f5d8b9c65a30536113b44c5d73b83d723fa208fbcd943475edd82"}
Sep 17 17:41:02.362261 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.362220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" event={"ID":"c3415d80-dec7-47e2-aded-58c3bd05537f","Type":"ContainerStarted","Data":"09108d49474b1b052db908a7d1b110cda9fca933bf8c4c00bafc2035ff3752ad"}
Sep 17 17:41:02.363247 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.363222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7zjm" event={"ID":"0868d5ee-d875-45d7-866e-5678b9f1411d","Type":"ContainerStarted","Data":"c944cf560eafa558073644c78d66b3f30f1475840f48d3e75bce7ef58f148a64"}
Sep 17 17:41:02.879443 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.879333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:02.879443 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.879422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:02.879658 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.879558    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:02.879658 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.879627    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs podName:06e9329e-dfb9-4ca8-9807-995754f7f505 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:04.879605654 +0000 UTC m=+7.715374907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs") pod "network-metrics-daemon-x9dql" (UID: "06e9329e-dfb9-4ca8-9807-995754f7f505") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:02.880056 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.880036    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:02.880121 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.880097    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret podName:45898ed0-e311-492a-ab02-7851051c85a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:04.880082376 +0000 UTC m=+7.715851582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret") pod "global-pull-secret-syncer-v47zx" (UID: "45898ed0-e311-492a-ab02-7851051c85a9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:02.979823 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:02.979776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:02.980005 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.979955    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:02.980005 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.979978    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:02.980005 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.979990    2565 projected.go:196] Error preparing data for projected volume kube-api-access-27rpg for pod openshift-network-diagnostics/network-check-target-5gtth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:02.980161 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:02.980050    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg podName:d1b48b76-797b-4760-9783-3a1b06ea75b7 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:04.98003204 +0000 UTC m=+7.815801256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-27rpg" (UniqueName: "kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg") pod "network-check-target-5gtth" (UID: "d1b48b76-797b-4760-9783-3a1b06ea75b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:03.313547 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:03.313363    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:01 +0000 UTC" deadline="2028-03-03 07:22:03.234597016 +0000 UTC"
Sep 17 17:41:03.313547 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:03.313442    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12781h40m59.921158918s"
Sep 17 17:41:03.346575 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:03.346546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:03.346750 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:03.346687    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:03.376422 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:03.376054    2565 generic.go:356] "Generic (PLEG): container finished" podID="15411fab6b97a438488300fab1eaf100" containerID="57cdf261517fd7c1e767a232458e437ef073efb1540d56f117604873c4ea6d3e" exitCode=0
Sep 17 17:41:03.376422 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:03.376129    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal" event={"ID":"15411fab6b97a438488300fab1eaf100","Type":"ContainerDied","Data":"57cdf261517fd7c1e767a232458e437ef073efb1540d56f117604873c4ea6d3e"}
Sep 17 17:41:03.391744 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:03.391675    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal" podStartSLOduration=3.391653685 podStartE2EDuration="3.391653685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:02.387266295 +0000 UTC m=+5.223035636" watchObservedRunningTime="2026-09-17 17:41:03.391653685 +0000 UTC m=+6.227422912"
Sep 17 17:41:04.343623 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:04.343590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:04.344067 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.343725    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:04.344067 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:04.344062    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:04.344180 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.344157    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:04.416565 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:04.416529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal" event={"ID":"15411fab6b97a438488300fab1eaf100","Type":"ContainerStarted","Data":"b551e0d503ed5d66df177f4efafdd08a82d277f5acc87e72c65b21c527641757"}
Sep 17 17:41:04.895318 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:04.895283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:04.895506 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:04.895335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:04.895506 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.895489    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.895740 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.895552    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs podName:06e9329e-dfb9-4ca8-9807-995754f7f505 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:08.895532852 +0000 UTC m=+11.731302073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs") pod "network-metrics-daemon-x9dql" (UID: "06e9329e-dfb9-4ca8-9807-995754f7f505") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.895972 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.895952    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.896058 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.896014    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret podName:45898ed0-e311-492a-ab02-7851051c85a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:08.895995713 +0000 UTC m=+11.731764930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret") pod "global-pull-secret-syncer-v47zx" (UID: "45898ed0-e311-492a-ab02-7851051c85a9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.996208 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:04.996175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:04.996372 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.996337    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:04.996372 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.996370    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:04.996516 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.996382    2565 projected.go:196] Error preparing data for projected volume kube-api-access-27rpg for pod openshift-network-diagnostics/network-check-target-5gtth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:04.996516 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:04.996463    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg podName:d1b48b76-797b-4760-9783-3a1b06ea75b7 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:08.996445081 +0000 UTC m=+11.832214298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-27rpg" (UniqueName: "kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg") pod "network-check-target-5gtth" (UID: "d1b48b76-797b-4760-9783-3a1b06ea75b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:05.344016 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:05.343497    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:05.344016 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:05.343636    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:06.343369 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:06.343337    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:06.343548 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:06.343489    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:06.343877 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:06.343859    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:06.343964 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:06.343946    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:07.344058 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:07.344019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:07.344521 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:07.344180    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:08.343865 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:08.343823    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:08.343865 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:08.343853    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:08.344141 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:08.343968    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:08.344141 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:08.344101    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:08.929297 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:08.929265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:08.929488 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:08.929315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:08.929488 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:08.929444    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:08.929488 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:08.929467    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:08.929666 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:08.929511    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret podName:45898ed0-e311-492a-ab02-7851051c85a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:16.929488367 +0000 UTC m=+19.765257588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret") pod "global-pull-secret-syncer-v47zx" (UID: "45898ed0-e311-492a-ab02-7851051c85a9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:08.929666 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:08.929532    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs podName:06e9329e-dfb9-4ca8-9807-995754f7f505 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:16.929522315 +0000 UTC m=+19.765291516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs") pod "network-metrics-daemon-x9dql" (UID: "06e9329e-dfb9-4ca8-9807-995754f7f505") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:09.029725 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:09.029676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:09.029917 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:09.029884    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:09.029917 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:09.029908    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:09.030026 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:09.029920    2565 projected.go:196] Error preparing data for projected volume kube-api-access-27rpg for pod openshift-network-diagnostics/network-check-target-5gtth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:09.030026 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:09.029977    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg podName:d1b48b76-797b-4760-9783-3a1b06ea75b7 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:17.029959366 +0000 UTC m=+19.865728584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-27rpg" (UniqueName: "kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg") pod "network-check-target-5gtth" (UID: "d1b48b76-797b-4760-9783-3a1b06ea75b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:09.344679 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:09.344126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:09.344679 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:09.344252    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:10.344049 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:10.344015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:10.344194 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:10.344059    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:10.344194 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:10.344154    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:10.344307 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:10.344274    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:11.344447 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:11.344317    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:11.344892 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:11.344485    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:12.343878 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:12.343848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:12.343982 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:12.343855    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:12.343982 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:12.343953    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:12.344063 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:12.344012    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:13.343801 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:13.343753    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:13.344292 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:13.343883    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:14.344002 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:14.343972    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:14.344463 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:14.343975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:14.344463 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:14.344083    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:14.344463 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:14.344181    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:15.343859 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:15.343822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:15.344021 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:15.343962    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:16.343209 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:16.343171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:16.343643 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:16.343171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:16.343643 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:16.343284    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:16.343643 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:16.343416    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:16.989818 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:16.989779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:16.989818 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:16.989821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:16.990037 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:16.989940    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:16.990037 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:16.990003    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:16.990158 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:16.990006    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret podName:45898ed0-e311-492a-ab02-7851051c85a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:32.989992599 +0000 UTC m=+35.825761801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret") pod "global-pull-secret-syncer-v47zx" (UID: "45898ed0-e311-492a-ab02-7851051c85a9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:16.990158 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:16.990064    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs podName:06e9329e-dfb9-4ca8-9807-995754f7f505 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:32.990049335 +0000 UTC m=+35.825818549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs") pod "network-metrics-daemon-x9dql" (UID: "06e9329e-dfb9-4ca8-9807-995754f7f505") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:17.090745 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:17.090709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:17.090910 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:17.090894    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:17.090979 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:17.090915    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:17.090979 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:17.090924    2565 projected.go:196] Error preparing data for projected volume kube-api-access-27rpg for pod openshift-network-diagnostics/network-check-target-5gtth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:17.090979 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:17.090975    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg podName:d1b48b76-797b-4760-9783-3a1b06ea75b7 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.090962364 +0000 UTC m=+35.926731565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-27rpg" (UniqueName: "kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg") pod "network-check-target-5gtth" (UID: "d1b48b76-797b-4760-9783-3a1b06ea75b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:17.344290 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:17.344201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:17.344729 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:17.344349    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:18.343953 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:18.343919    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:18.343953 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:18.343938    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:18.344186 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:18.344049    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:18.344243 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:18.344189    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:19.343760 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:19.343732    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:19.344088 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:19.343819    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:20.343241 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.343130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:20.343327 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.343183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:20.343389 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:20.343359    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:20.343456 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:20.343411    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:20.451741 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.451709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lpzgv" event={"ID":"94f278e6-8be6-42ba-b87a-be298b200738","Type":"ContainerStarted","Data":"ee9c90a31fd227e69fd28d6305e803509a0da898432b1474d9ad961531476c57"}
Sep 17 17:41:20.453553 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.453521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2vkjk" event={"ID":"fab829e2-cb5a-4779-bff8-0a6ae8a78c3f","Type":"ContainerStarted","Data":"439bcc83bdda86f6ab65fc19395ba2ac01153c138bf03b4f96cc86b52408127f"}
Sep 17 17:41:20.455315 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.455294    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3415d80-dec7-47e2-aded-58c3bd05537f" containerID="a307b9c211a6b7a2db26095cdd47d32ea78b56aff136cf8d80147e933409c34b" exitCode=0
Sep 17 17:41:20.455435 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.455356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" event={"ID":"c3415d80-dec7-47e2-aded-58c3bd05537f","Type":"ContainerDied","Data":"a307b9c211a6b7a2db26095cdd47d32ea78b56aff136cf8d80147e933409c34b"}
Sep 17 17:41:20.457098 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.457073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7zjm" event={"ID":"0868d5ee-d875-45d7-866e-5678b9f1411d","Type":"ContainerStarted","Data":"fc5c70d2055ca247e5ee50dff4a820763d589f44d4973f8934d6399df261e2a1"}
Sep 17 17:41:20.462632 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.462589    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bvd5m" event={"ID":"1ce68719-e9b2-4fee-b1d2-562bd57bca45","Type":"ContainerStarted","Data":"83e8fda80f3e10bd7880a4551dcd7c39c24a15447b42309f8d121f0091849e02"}
Sep 17 17:41:20.465091 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.465066    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:41:20.465421 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.465375    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef7c1c2f-a805-434a-9537-bed7df1e3570" containerID="7eda3caaddbbdb06f640dbff53c1710b8a76cf0cbe4fb9cc662c58ec076053dd" exitCode=1
Sep 17 17:41:20.465501 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.465440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"95f5f8e6bca0eb40bd75dc970c29daa4b3382f12306ba6908266ceb3e30642c6"}
Sep 17 17:41:20.465565 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.465500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"9757bd2f39a8b7ff5dd80d68945ac2d1f70346d91287d8c3cbaf21f630457f72"}
Sep 17 17:41:20.465565 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.465550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerDied","Data":"7eda3caaddbbdb06f640dbff53c1710b8a76cf0cbe4fb9cc662c58ec076053dd"}
Sep 17 17:41:20.465664 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.465598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"debda597399d6003e82c636c52216df60427e752ec1fc082b09569c99803990a"}
Sep 17 17:41:20.467007 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.466985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kbdfg" event={"ID":"3415e108-2762-4d68-9cdc-10f826382faf","Type":"ContainerStarted","Data":"aba9da4ff0f4dec3f5fec8a38901b957ce3cec96cf0e66214aba50d31052008b"}
Sep 17 17:41:20.468640 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.468607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz" event={"ID":"16cd028c-cfdd-4f39-8fd2-647ddad31d62","Type":"ContainerStarted","Data":"b344c5b5dd245f0e782ff90b4dfd39551288da074661f25e941eb1c2bb0c35c4"}
Sep 17 17:41:20.469718 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.469683    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal" podStartSLOduration=20.469671088 podStartE2EDuration="20.469671088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:04.434521361 +0000 UTC m=+7.270290587" watchObservedRunningTime="2026-09-17 17:41:20.469671088 +0000 UTC m=+23.305440311"
Sep 17 17:41:20.470098 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.470068    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lpzgv" podStartSLOduration=3.871220426 podStartE2EDuration="21.470060767s" totalImagesPullingTime="17.598840341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:20.469432801 +0000 UTC m=+23.305202025" watchObservedRunningTime="2026-09-17 17:41:20.470060767 +0000 UTC m=+23.305830005"
Sep 17 17:41:20.503616 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.503577    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2vkjk" podStartSLOduration=3.993282457 podStartE2EDuration="21.503558607s" totalImagesPullingTime="17.51027615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:20.503314194 +0000 UTC m=+23.339083417" watchObservedRunningTime="2026-09-17 17:41:20.503558607 +0000 UTC m=+23.339327832"
Sep 17 17:41:20.518588 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.518544    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bvd5m" podStartSLOduration=3.978775948 podStartE2EDuration="21.518525929s" totalImagesPullingTime="17.539749981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:20.518045395 +0000 UTC m=+23.353814620" watchObservedRunningTime="2026-09-17 17:41:20.518525929 +0000 UTC m=+23.354295151"
Sep 17 17:41:20.534192 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.534149    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s7zjm" podStartSLOduration=8.21286137 podStartE2EDuration="21.534131297s" totalImagesPullingTime="13.321269927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:20.533888974 +0000 UTC m=+23.369658198" watchObservedRunningTime="2026-09-17 17:41:20.534131297 +0000 UTC m=+23.369900521"
Sep 17 17:41:20.550438 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:20.550367    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kbdfg" podStartSLOduration=4.001055949 podStartE2EDuration="21.550350279s" totalImagesPullingTime="17.54929433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:20.550037792 +0000 UTC m=+23.385807014" watchObservedRunningTime="2026-09-17 17:41:20.550350279 +0000 UTC m=+23.386119503"
Sep 17 17:41:21.305068 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.304862    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:41:21.322541 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.322467    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:41:21.305053586Z","UUID":"f6ab3b1b-24de-47dd-822f-11772c365b6e","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:41:21.324218 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.324186    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:41:21.324322 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.324224    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:41:21.343449 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.343427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:21.343588 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:21.343567    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:21.472249 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.472216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wmcq4" event={"ID":"172b0396-6b46-453f-b279-c37f1b0cc403","Type":"ContainerStarted","Data":"1e07fed69505961a651feb7b0d888a3cfff60791fbabb5985a7fcfaa68a5feba"}
Sep 17 17:41:21.475704 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.475677    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:41:21.476103 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.476079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"dba2a0f702f516e47ca56c00f69cd0ec4fa237fca43e9a8707c56dfe09854585"}
Sep 17 17:41:21.476216 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.476149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"07bae31a33436550c9aee58b6ded589cd11f28a022a8e765516563612cf4ff06"}
Sep 17 17:41:21.479361 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.478460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz" event={"ID":"16cd028c-cfdd-4f39-8fd2-647ddad31d62","Type":"ContainerStarted","Data":"973e371e6d75b4c6b1de698fa8c2e42f93609eacebf9f38fc42497a65fe7b089"}
Sep 17 17:41:21.487798 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:21.487751    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wmcq4" podStartSLOduration=4.9739638809999995 podStartE2EDuration="22.4877333s" totalImagesPullingTime="17.513769419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:21.487570531 +0000 UTC m=+24.323339755" watchObservedRunningTime="2026-09-17 17:41:21.4877333 +0000 UTC m=+24.323502744"
Sep 17 17:41:22.343832 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:22.343804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:22.343832 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:22.343823    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:22.344069 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:22.343900    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:22.344069 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:22.344034    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:22.484177 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:22.484142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz" event={"ID":"16cd028c-cfdd-4f39-8fd2-647ddad31d62","Type":"ContainerStarted","Data":"94e463065312a3d5a44abdcd3da978ff571f11a750b630bc9db756132003fda3"}
Sep 17 17:41:23.078699 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:23.078662    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:23.079623 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:23.079601    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:23.343576 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:23.343497    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:23.343715 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:23.343634    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:23.489404 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:23.489362    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:41:23.489838 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:23.489816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"1a8f28e27c561d31675cd0ff9002a5b42fe4fc172f70093739312d0323f35b4a"}
Sep 17 17:41:23.490085 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:23.490070    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:23.490582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:23.490558    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s7zjm"
Sep 17 17:41:23.509522 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:23.509483    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5gz" podStartSLOduration=4.732586837 podStartE2EDuration="24.509470606s" totalImagesPullingTime="19.776883769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:23.509356122 +0000 UTC m=+26.345125347" watchObservedRunningTime="2026-09-17 17:41:23.509470606 +0000 UTC m=+26.345239828"
Sep 17 17:41:24.344085 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:24.344043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:24.344242 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:24.344044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:24.344242 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:24.344151    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:24.344322 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:24.344237    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:25.344265 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:25.344099    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:25.344808 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:25.344342    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:25.497217 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:25.497183    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:41:25.497599 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:25.497569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"e24e522ebbb0e3a9024e7c6786a8d9227a756a6675827237e39ec9455b55dd77"}
Sep 17 17:41:25.499938 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:25.497970    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:25.499938 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:25.498039    2565 scope.go:117] "RemoveContainer" containerID="7eda3caaddbbdb06f640dbff53c1710b8a76cf0cbe4fb9cc662c58ec076053dd"
Sep 17 17:41:25.502499 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:25.502474    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3415d80-dec7-47e2-aded-58c3bd05537f" containerID="05403d3ea6693fbd1e1b42f3596d01ba9b7031d3b63c9103866a1584befc5759" exitCode=0
Sep 17 17:41:25.502593 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:25.502548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" event={"ID":"c3415d80-dec7-47e2-aded-58c3bd05537f","Type":"ContainerDied","Data":"05403d3ea6693fbd1e1b42f3596d01ba9b7031d3b63c9103866a1584befc5759"}
Sep 17 17:41:25.515019 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:25.515000    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:26.344131 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.344095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:26.344131 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.344114    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:26.344802 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:26.344217    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:26.344802 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:26.344322    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:26.509137 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.509113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:41:26.509499 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.509479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" event={"ID":"ef7c1c2f-a805-434a-9537-bed7df1e3570","Type":"ContainerStarted","Data":"20766cfdff2460f769e79e862d187645ccb4e5098395fa960c8e00ad1a48d1a7"}
Sep 17 17:41:26.509593 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.509578    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:41:26.509850 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.509831    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:26.525257 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.525234    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:26.560358 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.560311    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jk425" podStartSLOduration=10.017323319 podStartE2EDuration="27.560297886s" totalImagesPullingTime="17.542974567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:26.559862393 +0000 UTC m=+29.395631617" watchObservedRunningTime="2026-09-17 17:41:26.560297886 +0000 UTC m=+29.396067108"
Sep 17 17:41:26.864278 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.864117    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gtth"]
Sep 17 17:41:26.864442 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.864374    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:26.864494 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:26.864474    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:26.864870 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.864851    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x9dql"]
Sep 17 17:41:26.864952 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.864942    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:26.865030 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:26.865015    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:26.876325 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.876271    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v47zx"]
Sep 17 17:41:26.876437 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:26.876356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:26.876489 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:26.876446    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:27.395840 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:27.395809    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:41:27.513705 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:27.513672    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3415d80-dec7-47e2-aded-58c3bd05537f" containerID="83098fcb60c7b1b6d0898d22ae8175cd2a3153d7245f9736809fcc0c114a1853" exitCode=0
Sep 17 17:41:27.513827 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:27.513761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" event={"ID":"c3415d80-dec7-47e2-aded-58c3bd05537f","Type":"ContainerDied","Data":"83098fcb60c7b1b6d0898d22ae8175cd2a3153d7245f9736809fcc0c114a1853"}
Sep 17 17:41:28.343617 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:28.343584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:28.343803 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:28.343674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:28.343803 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:28.343675    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:28.343803 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:28.343679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:28.343968 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:28.343796    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:28.343968 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:28.343891    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:29.521549 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:29.521517    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3415d80-dec7-47e2-aded-58c3bd05537f" containerID="97c8467f2a592738b1a6c817c3944e9d4ad1d0f3aa29b4888f0fb0a493ea3d01" exitCode=0
Sep 17 17:41:29.521900 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:29.521580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" event={"ID":"c3415d80-dec7-47e2-aded-58c3bd05537f","Type":"ContainerDied","Data":"97c8467f2a592738b1a6c817c3944e9d4ad1d0f3aa29b4888f0fb0a493ea3d01"}
Sep 17 17:41:30.343388 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:30.343357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:30.343388 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:30.343379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:30.343693 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:30.343379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:30.343693 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:30.343501    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:30.343693 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:30.343577    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:30.343693 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:30.343673    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:32.343450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.343411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:32.343450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.343448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:32.343977 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.343453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:32.343977 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:32.343526    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v47zx" podUID="45898ed0-e311-492a-ab02-7851051c85a9"
Sep 17 17:41:32.343977 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:32.343639    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9dql" podUID="06e9329e-dfb9-4ca8-9807-995754f7f505"
Sep 17 17:41:32.343977 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:32.343722    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gtth" podUID="d1b48b76-797b-4760-9783-3a1b06ea75b7"
Sep 17 17:41:32.698469 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.698437    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeReady"
Sep 17 17:41:32.698688 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.698577    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:41:32.739170 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.739145    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-645b995f97-6746s"]
Sep 17 17:41:32.763765 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.763729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-645b995f97-6746s"]
Sep 17 17:41:32.763905 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.763874    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.766085 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.766051    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:41:32.766296 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.766281    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:41:32.766379 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.766280    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mkt75\""
Sep 17 17:41:32.766379 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.766320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:41:32.773736 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.773704    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:41:32.783235 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.783216    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-snzb8"]
Sep 17 17:41:32.794312 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.794289    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6nx7f"]
Sep 17 17:41:32.794461 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.794445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-snzb8"
Sep 17 17:41:32.796672 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.796651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:41:32.796768 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.796691    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:41:32.797163 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.797116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xhmjd\""
Sep 17 17:41:32.797163 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.797131    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:41:32.806755 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.806737    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:32.808422 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.808330    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6nx7f"]
Sep 17 17:41:32.811581 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.811302    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n5jl\""
Sep 17 17:41:32.811581 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.811489    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:41:32.811581 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.811511    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:41:32.812160 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.812139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-snzb8"]
Sep 17 17:41:32.917302 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9rf\" (UniqueName: \"kubernetes.io/projected/93732b61-bff8-483a-bf62-a0a298041e71-kube-api-access-cj9rf\") pod \"ingress-canary-snzb8\" (UID: \"93732b61-bff8-483a-bf62-a0a298041e71\") " pod="openshift-ingress-canary/ingress-canary-snzb8"
Sep 17 17:41:32.917490 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93732b61-bff8-483a-bf62-a0a298041e71-cert\") pod \"ingress-canary-snzb8\" (UID: \"93732b61-bff8-483a-bf62-a0a298041e71\") " pod="openshift-ingress-canary/ingress-canary-snzb8"
Sep 17 17:41:32.917490 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-image-registry-private-configuration\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.917490 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.917490 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-registry-certificates\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.917706 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdw4\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-kube-api-access-czdw4\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.917706 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917548    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:32.917706 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-tmp-dir\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:32.917706 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b093799-bfd9-46a6-ae5f-181eab889604-ca-trust-extracted\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.917706 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5sm\" (UniqueName: \"kubernetes.io/projected/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-kube-api-access-cm5sm\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:32.917932 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-installation-pull-secrets\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.917932 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-bound-sa-token\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.917932 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-trusted-ca\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:32.917932 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:32.917816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-config-volume\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.018695 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-image-registry-private-configuration\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.018695 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.018695 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-registry-certificates\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdw4\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-kube-api-access-czdw4\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-tmp-dir\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b093799-bfd9-46a6-ae5f-181eab889604-ca-trust-extracted\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5sm\" (UniqueName: \"kubernetes.io/projected/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-kube-api-access-cm5sm\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-installation-pull-secrets\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-bound-sa-token\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.018972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-trusted-ca\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.019423 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.018979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-config-volume\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.019423 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.019029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9rf\" (UniqueName: \"kubernetes.io/projected/93732b61-bff8-483a-bf62-a0a298041e71-kube-api-access-cj9rf\") pod \"ingress-canary-snzb8\" (UID: \"93732b61-bff8-483a-bf62-a0a298041e71\") " pod="openshift-ingress-canary/ingress-canary-snzb8"
Sep 17 17:41:33.019423 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.019056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93732b61-bff8-483a-bf62-a0a298041e71-cert\") pod \"ingress-canary-snzb8\" (UID: \"93732b61-bff8-483a-bf62-a0a298041e71\") " pod="openshift-ingress-canary/ingress-canary-snzb8"
Sep 17 17:41:33.019423 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.019102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-tmp-dir\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.019423 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.019415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-registry-certificates\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.019696 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.018848    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:33.019696 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.019522    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645b995f97-6746s: secret "image-registry-tls" not found
Sep 17 17:41:33.019696 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.019573    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls podName:1b093799-bfd9-46a6-ae5f-181eab889604 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.519556387 +0000 UTC m=+36.355325600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls") pod "image-registry-645b995f97-6746s" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604") : secret "image-registry-tls" not found
Sep 17 17:41:33.019696 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.019458    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:33.019696 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.019614    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls podName:96fa9c3b-e4e8-446d-984b-b78f26e4b8b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.519605114 +0000 UTC m=+36.355374322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls") pod "dns-default-6nx7f" (UID: "96fa9c3b-e4e8-446d-984b-b78f26e4b8b3") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:33.019901 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.019504    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:33.019901 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.019707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b093799-bfd9-46a6-ae5f-181eab889604-ca-trust-extracted\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.019901 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.019734    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs podName:06e9329e-dfb9-4ca8-9807-995754f7f505 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:05.019722742 +0000 UTC m=+67.855491950 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs") pod "network-metrics-daemon-x9dql" (UID: "06e9329e-dfb9-4ca8-9807-995754f7f505") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:33.019901 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.019790    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:33.019901 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.019847    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret podName:45898ed0-e311-492a-ab02-7851051c85a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:05.019826324 +0000 UTC m=+67.855595540 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret") pod "global-pull-secret-syncer-v47zx" (UID: "45898ed0-e311-492a-ab02-7851051c85a9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:33.020511 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.020334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-config-volume\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.022076 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.021992    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-trusted-ca\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.024006 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.023971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-installation-pull-secrets\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.026491 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.026465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-image-registry-private-configuration\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.026600 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.026545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93732b61-bff8-483a-bf62-a0a298041e71-cert\") pod \"ingress-canary-snzb8\" (UID: \"93732b61-bff8-483a-bf62-a0a298041e71\") " pod="openshift-ingress-canary/ingress-canary-snzb8"
Sep 17 17:41:33.044514 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.038048    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fj7nq"]
Sep 17 17:41:33.060791 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.060762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5sm\" (UniqueName: \"kubernetes.io/projected/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-kube-api-access-cm5sm\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.060911 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.060804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-bound-sa-token\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.060911 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.060817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9rf\" (UniqueName: \"kubernetes.io/projected/93732b61-bff8-483a-bf62-a0a298041e71-kube-api-access-cj9rf\") pod \"ingress-canary-snzb8\" (UID: \"93732b61-bff8-483a-bf62-a0a298041e71\") " pod="openshift-ingress-canary/ingress-canary-snzb8"
Sep 17 17:41:33.070558 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.070537    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdw4\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-kube-api-access-czdw4\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.079381 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.079360    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.081590 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.081566    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fj7nq"]
Sep 17 17:41:33.082647 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.082627    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:41:33.083216 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.083003    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zrqgb\""
Sep 17 17:41:33.083216 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.083061    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:41:33.083216 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.083067    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:41:33.083452 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.083351    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:41:33.103557 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.103533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-snzb8"
Sep 17 17:41:33.119554 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.119528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:33.119710 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.119657    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:33.119710 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.119681    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:33.119710 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.119691    2565 projected.go:196] Error preparing data for projected volume kube-api-access-27rpg for pod openshift-network-diagnostics/network-check-target-5gtth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:33.119872 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.119743    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg podName:d1b48b76-797b-4760-9783-3a1b06ea75b7 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:05.119726122 +0000 UTC m=+67.955495339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-27rpg" (UniqueName: "kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg") pod "network-check-target-5gtth" (UID: "d1b48b76-797b-4760-9783-3a1b06ea75b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:33.220119 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.220067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.220295 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.220150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.220355 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.220290    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-data-volume\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.220355 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.220331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvbz\" (UniqueName: \"kubernetes.io/projected/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-api-access-7rvbz\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.220493 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.220366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-crio-socket\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.321434 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.321339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.321597 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.321512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-data-volume\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.321597 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.321516    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:33.321597 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.321532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvbz\" (UniqueName: \"kubernetes.io/projected/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-api-access-7rvbz\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.321597 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.321549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-crio-socket\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.321597 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.321585    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.821570096 +0000 UTC m=+36.657339297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:33.321911 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.321600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.321911 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.321686    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:33.321911 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.321743    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.821730387 +0000 UTC m=+36.657499589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:33.321911 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.321762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-crio-socket\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.322038 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.321955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-data-volume\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.331101 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.331079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvbz\" (UniqueName: \"kubernetes.io/projected/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-api-access-7rvbz\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.466057 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.466030    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748"]
Sep 17 17:41:33.479886 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.479862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748"
Sep 17 17:41:33.480616 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.480592    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748"]
Sep 17 17:41:33.482248 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.482224    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 17:41:33.482359 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.482290    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5tk4l\""
Sep 17 17:41:33.482854 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.482833    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:33.524239 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.524081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:33.524363 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.524267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:33.524363 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.524271    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:33.524363 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.524321    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645b995f97-6746s: secret "image-registry-tls" not found
Sep 17 17:41:33.524543 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.524376    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:33.524543 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.524384    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls podName:1b093799-bfd9-46a6-ae5f-181eab889604 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:34.524366659 +0000 UTC m=+37.360135876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls") pod "image-registry-645b995f97-6746s" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604") : secret "image-registry-tls" not found
Sep 17 17:41:33.524543 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.524525    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls podName:96fa9c3b-e4e8-446d-984b-b78f26e4b8b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:34.524500794 +0000 UTC m=+37.360270011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls") pod "dns-default-6nx7f" (UID: "96fa9c3b-e4e8-446d-984b-b78f26e4b8b3") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:33.624976 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.624941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ssfm\" (UniqueName: \"kubernetes.io/projected/01a47f0c-e831-4b54-b4c8-6171316436bb-kube-api-access-2ssfm\") pod \"migrator-76794cbd6c-h4748\" (UID: \"01a47f0c-e831-4b54-b4c8-6171316436bb\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748"
Sep 17 17:41:33.725576 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.725547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssfm\" (UniqueName: \"kubernetes.io/projected/01a47f0c-e831-4b54-b4c8-6171316436bb-kube-api-access-2ssfm\") pod \"migrator-76794cbd6c-h4748\" (UID: \"01a47f0c-e831-4b54-b4c8-6171316436bb\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748"
Sep 17 17:41:33.734367 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.734340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssfm\" (UniqueName: \"kubernetes.io/projected/01a47f0c-e831-4b54-b4c8-6171316436bb-kube-api-access-2ssfm\") pod \"migrator-76794cbd6c-h4748\" (UID: \"01a47f0c-e831-4b54-b4c8-6171316436bb\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748"
Sep 17 17:41:33.788730 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.788703    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748"
Sep 17 17:41:33.794427 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.794379    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-snzb8"]
Sep 17 17:41:33.799481 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:33.799454    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93732b61_bff8_483a_bf62_a0a298041e71.slice/crio-55f5f17f3c80d391f6d7bba556a1ea473312d4ca4cb831b031e14baeb3149916 WatchSource:0}: Error finding container 55f5f17f3c80d391f6d7bba556a1ea473312d4ca4cb831b031e14baeb3149916: Status 404 returned error can't find the container with id 55f5f17f3c80d391f6d7bba556a1ea473312d4ca4cb831b031e14baeb3149916
Sep 17 17:41:33.826650 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.826584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.826650 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:33.826637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:33.826816 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.826710    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:33.826816 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.826777    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:34.826762214 +0000 UTC m=+37.662531415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:33.826816 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.826783    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:33.826933 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:33.826834    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:34.826818676 +0000 UTC m=+37.662587893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:34.079712 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.079652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748"]
Sep 17 17:41:34.102355 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:34.102329    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a47f0c_e831_4b54_b4c8_6171316436bb.slice/crio-61db705abcc9509679886bf512aa3ca1f493027874ef3fc710370535e1511142 WatchSource:0}: Error finding container 61db705abcc9509679886bf512aa3ca1f493027874ef3fc710370535e1511142: Status 404 returned error can't find the container with id 61db705abcc9509679886bf512aa3ca1f493027874ef3fc710370535e1511142
Sep 17 17:41:34.344128 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.344053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:41:34.344306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.344141    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:41:34.344306 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.344169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:41:34.346435 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.346413    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7j64\""
Sep 17 17:41:34.346880 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.346865    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:41:34.346965 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.346879    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:41:34.346965 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.346920    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:41:34.347071 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.346995    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:41:34.347326 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.347313    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ccxk9\""
Sep 17 17:41:34.484910 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.484878    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bvd5m_1ce68719-e9b2-4fee-b1d2-562bd57bca45/dns-node-resolver/0.log"
Sep 17 17:41:34.532229 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.532200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:34.532384 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.532248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:34.532384 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.532374    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:34.532526 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.532452    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:34.532526 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.532466    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645b995f97-6746s: secret "image-registry-tls" not found
Sep 17 17:41:34.532526 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.532507    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls podName:96fa9c3b-e4e8-446d-984b-b78f26e4b8b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.532488993 +0000 UTC m=+39.368258207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls") pod "dns-default-6nx7f" (UID: "96fa9c3b-e4e8-446d-984b-b78f26e4b8b3") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:34.532526 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.532524    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls podName:1b093799-bfd9-46a6-ae5f-181eab889604 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.532516374 +0000 UTC m=+39.368285574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls") pod "image-registry-645b995f97-6746s" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604") : secret "image-registry-tls" not found
Sep 17 17:41:34.536022 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.535990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748" event={"ID":"01a47f0c-e831-4b54-b4c8-6171316436bb","Type":"ContainerStarted","Data":"61db705abcc9509679886bf512aa3ca1f493027874ef3fc710370535e1511142"}
Sep 17 17:41:34.537380 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.537352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-snzb8" event={"ID":"93732b61-bff8-483a-bf62-a0a298041e71","Type":"ContainerStarted","Data":"55f5f17f3c80d391f6d7bba556a1ea473312d4ca4cb831b031e14baeb3149916"}
Sep 17 17:41:34.540547 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.540523    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3415d80-dec7-47e2-aded-58c3bd05537f" containerID="5c1454b5ba8794f5d7ba1a7d1d8bf08396129dc7a4566cde7a6c9876741d2a70" exitCode=0
Sep 17 17:41:34.540636 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.540564    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" event={"ID":"c3415d80-dec7-47e2-aded-58c3bd05537f","Type":"ContainerDied","Data":"5c1454b5ba8794f5d7ba1a7d1d8bf08396129dc7a4566cde7a6c9876741d2a70"}
Sep 17 17:41:34.836320 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.836245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:34.836559 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.836372    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:34.836559 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.836465    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.836444401 +0000 UTC m=+39.672213616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:34.836758 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:34.836730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:34.836887 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.836871    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:34.836946 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:34.836926    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.83691018 +0000 UTC m=+39.672679387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:35.030154 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.030114    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-mpm5b"]
Sep 17 17:41:35.049881 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.049835    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-mpm5b"]
Sep 17 17:41:35.050045 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.049974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.052573 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.052358    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 17:41:35.052573 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.052357    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 17:41:35.052573 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.052383    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 17:41:35.052573 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.052462    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 17:41:35.052573 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.052480    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4lxrw\""
Sep 17 17:41:35.139742 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.139705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4249249b-dd38-4a26-9081-6d5c7d86b0a4-signing-cabundle\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.139910 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.139816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbwqc\" (UniqueName: \"kubernetes.io/projected/4249249b-dd38-4a26-9081-6d5c7d86b0a4-kube-api-access-vbwqc\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.139910 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.139859    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4249249b-dd38-4a26-9081-6d5c7d86b0a4-signing-key\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.240642 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.240605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4249249b-dd38-4a26-9081-6d5c7d86b0a4-signing-cabundle\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.240832 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.240718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwqc\" (UniqueName: \"kubernetes.io/projected/4249249b-dd38-4a26-9081-6d5c7d86b0a4-kube-api-access-vbwqc\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.240832 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.240758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4249249b-dd38-4a26-9081-6d5c7d86b0a4-signing-key\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.241461 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.241440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4249249b-dd38-4a26-9081-6d5c7d86b0a4-signing-cabundle\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.243848 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.243824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4249249b-dd38-4a26-9081-6d5c7d86b0a4-signing-key\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.249354 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.249310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbwqc\" (UniqueName: \"kubernetes.io/projected/4249249b-dd38-4a26-9081-6d5c7d86b0a4-kube-api-access-vbwqc\") pod \"service-ca-55f6fc7b86-mpm5b\" (UID: \"4249249b-dd38-4a26-9081-6d5c7d86b0a4\") " pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.284735 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.284678    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2vkjk_fab829e2-cb5a-4779-bff8-0a6ae8a78c3f/node-ca/0.log"
Sep 17 17:41:35.361287 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.361250    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b"
Sep 17 17:41:35.546067 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.545996    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3415d80-dec7-47e2-aded-58c3bd05537f" containerID="9ce4b84b66d08b2eb57faa1c8134a4282f7175ce9a806d2fb0e26cd37dfb1806" exitCode=0
Sep 17 17:41:35.546067 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.546042    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" event={"ID":"c3415d80-dec7-47e2-aded-58c3bd05537f","Type":"ContainerDied","Data":"9ce4b84b66d08b2eb57faa1c8134a4282f7175ce9a806d2fb0e26cd37dfb1806"}
Sep 17 17:41:35.677815 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:35.677782    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-mpm5b"]
Sep 17 17:41:35.681881 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:35.681856    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4249249b_dd38_4a26_9081_6d5c7d86b0a4.slice/crio-0e06fe51c6f87b13993983fc67d0971dc6751342bc0e72bddfd72e80eec92004 WatchSource:0}: Error finding container 0e06fe51c6f87b13993983fc67d0971dc6751342bc0e72bddfd72e80eec92004: Status 404 returned error can't find the container with id 0e06fe51c6f87b13993983fc67d0971dc6751342bc0e72bddfd72e80eec92004
Sep 17 17:41:36.551858 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:36.551832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" event={"ID":"c3415d80-dec7-47e2-aded-58c3bd05537f","Type":"ContainerStarted","Data":"d6f3281aa90d358797836bebe031d5a99b4f5e6f72a779bb11c2fa57aaefd37a"}
Sep 17 17:41:36.552443 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:36.552424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:36.552533 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:36.552462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:36.552533 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.552516    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:36.552533 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.552530    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645b995f97-6746s: secret "image-registry-tls" not found
Sep 17 17:41:36.552653 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.552555    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:36.552653 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.552583    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls podName:1b093799-bfd9-46a6-ae5f-181eab889604 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.552564841 +0000 UTC m=+43.388334062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls") pod "image-registry-645b995f97-6746s" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604") : secret "image-registry-tls" not found
Sep 17 17:41:36.552653 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.552604    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls podName:96fa9c3b-e4e8-446d-984b-b78f26e4b8b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.552588955 +0000 UTC m=+43.388358176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls") pod "dns-default-6nx7f" (UID: "96fa9c3b-e4e8-446d-984b-b78f26e4b8b3") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:36.553081 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:36.553058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b" event={"ID":"4249249b-dd38-4a26-9081-6d5c7d86b0a4","Type":"ContainerStarted","Data":"0e06fe51c6f87b13993983fc67d0971dc6751342bc0e72bddfd72e80eec92004"}
Sep 17 17:41:36.554505 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:36.554482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748" event={"ID":"01a47f0c-e831-4b54-b4c8-6171316436bb","Type":"ContainerStarted","Data":"920f3a7b02b1e756a1a6318c5e5abb8d16617796114a5c2e3c8bfffffb2e01c2"}
Sep 17 17:41:36.582072 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:36.582032    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zvpl4" podStartSLOduration=9.401556239 podStartE2EDuration="37.582019259s" totalImagesPullingTime="28.18046302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:36.581113321 +0000 UTC m=+39.416882541" watchObservedRunningTime="2026-09-17 17:41:36.582019259 +0000 UTC m=+39.417788480"
Sep 17 17:41:36.854656 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:36.854623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:36.854776 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:36.854688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:36.854776 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.854754    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:36.854884 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.854827    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.854804848 +0000 UTC m=+43.690574069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:36.854884 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.854831    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:36.854996 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:36.854893    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.854874696 +0000 UTC m=+43.690643911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:37.559406 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:37.559338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748" event={"ID":"01a47f0c-e831-4b54-b4c8-6171316436bb","Type":"ContainerStarted","Data":"35a67ec6eac8611c0994a757caae03ead8f39130874c7374b992ce63a2ef9548"}
Sep 17 17:41:37.561508 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:37.561484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-snzb8" event={"ID":"93732b61-bff8-483a-bf62-a0a298041e71","Type":"ContainerStarted","Data":"ecb0885a0ce06f03ea18aee0ee2df0bf902a0ec5d90855d618affce1dd2b2dca"}
Sep 17 17:41:37.577384 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:37.577341    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-h4748" podStartSLOduration=2.996462621 podStartE2EDuration="4.577326884s" totalImagesPullingTime="1.580864263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:37.576547139 +0000 UTC m=+40.412316374" watchObservedRunningTime="2026-09-17 17:41:37.577326884 +0000 UTC m=+40.413096135"
Sep 17 17:41:37.592750 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:37.592696    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-snzb8" podStartSLOduration=2.609986705 podStartE2EDuration="5.592680858s" totalImagesPullingTime="2.982694153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:37.59194746 +0000 UTC m=+40.427716684" watchObservedRunningTime="2026-09-17 17:41:37.592680858 +0000 UTC m=+40.428450082"
Sep 17 17:41:38.565865 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:38.565812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b" event={"ID":"4249249b-dd38-4a26-9081-6d5c7d86b0a4","Type":"ContainerStarted","Data":"bb2f0afbfc7f73d091d6c6f933d348224fbf77f249983f8bb6b52bbd76d1d177"}
Sep 17 17:41:38.582309 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:38.582262    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-mpm5b" podStartSLOduration=1.212442955 podStartE2EDuration="3.582246106s" totalImagesPullingTime="2.369803151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:38.581236169 +0000 UTC m=+41.417005412" watchObservedRunningTime="2026-09-17 17:41:38.582246106 +0000 UTC m=+41.418015348"
Sep 17 17:41:40.585602 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:40.585563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:40.585971 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.585700    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:40.585971 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.585730    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645b995f97-6746s: secret "image-registry-tls" not found
Sep 17 17:41:40.585971 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:40.585726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:40.585971 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.585860    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:40.585971 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.585890    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls podName:1b093799-bfd9-46a6-ae5f-181eab889604 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:48.58587106 +0000 UTC m=+51.421640268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls") pod "image-registry-645b995f97-6746s" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604") : secret "image-registry-tls" not found
Sep 17 17:41:40.585971 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.585910    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls podName:96fa9c3b-e4e8-446d-984b-b78f26e4b8b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:48.585901047 +0000 UTC m=+51.421670248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls") pod "dns-default-6nx7f" (UID: "96fa9c3b-e4e8-446d-984b-b78f26e4b8b3") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:40.887894 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:40.887817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:40.887894 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:40.887863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:40.888066 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.887937    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:40.888066 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.887974    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:40.888066 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.888002    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:48.887987675 +0000 UTC m=+51.723756876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:40.888066 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:40.888018    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:48.88800792 +0000 UTC m=+51.723777120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:48.651189 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.651155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:48.651655 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.651198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:48.654765 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.654742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96fa9c3b-e4e8-446d-984b-b78f26e4b8b3-metrics-tls\") pod \"dns-default-6nx7f\" (UID: \"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3\") " pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:48.654881 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.654794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"image-registry-645b995f97-6746s\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") " pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:48.676644 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.676624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:48.717445 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.717422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:48.954228 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.954194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:48.954383 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.954344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:48.954475 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:48.954457    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:48.954535 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:41:48.954524    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls podName:fbc275f3-5fe6-40d8-b267-5cc0c7e7e143 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:04.954508321 +0000 UTC m=+67.790277527 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fj7nq" (UID: "fbc275f3-5fe6-40d8-b267-5cc0c7e7e143") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:48.954925 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.954907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:41:48.996334 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:48.996307    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-645b995f97-6746s"]
Sep 17 17:41:49.000454 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:49.000429    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b093799_bfd9_46a6_ae5f_181eab889604.slice/crio-88bfa448fa81da518ac20e26c8d68612ef824546429157bfd2fce038f9a1b78f WatchSource:0}: Error finding container 88bfa448fa81da518ac20e26c8d68612ef824546429157bfd2fce038f9a1b78f: Status 404 returned error can't find the container with id 88bfa448fa81da518ac20e26c8d68612ef824546429157bfd2fce038f9a1b78f
Sep 17 17:41:49.028326 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:49.028301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6nx7f"]
Sep 17 17:41:49.032034 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:49.032006    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fa9c3b_e4e8_446d_984b_b78f26e4b8b3.slice/crio-4b97a41a2e1112e47fedf56fe605f9fe4c8672a9e3c2f4f09a3fba83b935d1ee WatchSource:0}: Error finding container 4b97a41a2e1112e47fedf56fe605f9fe4c8672a9e3c2f4f09a3fba83b935d1ee: Status 404 returned error can't find the container with id 4b97a41a2e1112e47fedf56fe605f9fe4c8672a9e3c2f4f09a3fba83b935d1ee
Sep 17 17:41:49.598006 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:49.597974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645b995f97-6746s" event={"ID":"1b093799-bfd9-46a6-ae5f-181eab889604","Type":"ContainerStarted","Data":"3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc"}
Sep 17 17:41:49.598195 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:49.598034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645b995f97-6746s" event={"ID":"1b093799-bfd9-46a6-ae5f-181eab889604","Type":"ContainerStarted","Data":"88bfa448fa81da518ac20e26c8d68612ef824546429157bfd2fce038f9a1b78f"}
Sep 17 17:41:49.598195 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:49.598062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:41:49.599439 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:49.599406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6nx7f" event={"ID":"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3","Type":"ContainerStarted","Data":"4b97a41a2e1112e47fedf56fe605f9fe4c8672a9e3c2f4f09a3fba83b935d1ee"}
Sep 17 17:41:49.621574 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:49.621523    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-645b995f97-6746s" podStartSLOduration=24.621506184 podStartE2EDuration="24.621506184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:49.620006118 +0000 UTC m=+52.455775342" watchObservedRunningTime="2026-09-17 17:41:49.621506184 +0000 UTC m=+52.457275407"
Sep 17 17:41:51.610213 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:51.610135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6nx7f" event={"ID":"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3","Type":"ContainerStarted","Data":"dca67436610a1d9a763d0f4b40e45927f666b64dfd742c3474ea0f00a5baeac7"}
Sep 17 17:41:51.610213 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:51.610189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6nx7f" event={"ID":"96fa9c3b-e4e8-446d-984b-b78f26e4b8b3","Type":"ContainerStarted","Data":"7e2e810796a64bd77bd19d94b56ec9a520fe2255345bf811617744c40490d7c5"}
Sep 17 17:41:51.610598 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:51.610229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:41:51.629920 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:51.629873    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6nx7f" podStartSLOduration=17.630645601 podStartE2EDuration="19.629859477s" totalImagesPullingTime="1.999213876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:51.628876058 +0000 UTC m=+54.464645282" watchObservedRunningTime="2026-09-17 17:41:51.629859477 +0000 UTC m=+54.465628695"
Sep 17 17:41:55.896284 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.896250    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"]
Sep 17 17:41:55.928004 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.927973    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"]
Sep 17 17:41:55.928152 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.928090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"
Sep 17 17:41:55.941869 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.941848    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"]
Sep 17 17:41:55.941985 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.941961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:55.951227 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.951205    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:41:55.953800 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.953777    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 17:41:55.953928 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.953850    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2fxhg\""
Sep 17 17:41:55.953988 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.953953    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 17:41:55.954087 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.954071    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:41:55.954168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.954140    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:41:55.954475 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.954461    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 17:41:55.954551 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.954491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:41:55.954551 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.954501    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 17:41:55.954857 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.954839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"]
Sep 17 17:41:55.976507 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:55.976485    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-645b995f97-6746s"]
Sep 17 17:41:56.008322 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.008302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gwt5\" (UniqueName: \"kubernetes.io/projected/b7258f31-b6f8-4047-a7cb-c97e3cb35e0f-kube-api-access-7gwt5\") pod \"managed-serviceaccount-addon-agent-7df488fbbd-smcmr\" (UID: \"b7258f31-b6f8-4047-a7cb-c97e3cb35e0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"
Sep 17 17:41:56.008422 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.008338    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fb8aed-02e5-47c9-a9a3-e5091da539fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.008422 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.008359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7258f31-b6f8-4047-a7cb-c97e3cb35e0f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7df488fbbd-smcmr\" (UID: \"b7258f31-b6f8-4047-a7cb-c97e3cb35e0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"
Sep 17 17:41:56.008422 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.008384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.008532 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.008450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ftz\" (UniqueName: \"kubernetes.io/projected/83fb8aed-02e5-47c9-a9a3-e5091da539fe-kube-api-access-f2ftz\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.008532 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.008480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.008532 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.008510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-ca\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.008532 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.008527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-hub\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.104518 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.104494    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-794f84759f-p2xmg"]
Sep 17 17:41:56.108811 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.108791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gwt5\" (UniqueName: \"kubernetes.io/projected/b7258f31-b6f8-4047-a7cb-c97e3cb35e0f-kube-api-access-7gwt5\") pod \"managed-serviceaccount-addon-agent-7df488fbbd-smcmr\" (UID: \"b7258f31-b6f8-4047-a7cb-c97e3cb35e0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"
Sep 17 17:41:56.108868 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.108831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fb8aed-02e5-47c9-a9a3-e5091da539fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.108868 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.108851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7258f31-b6f8-4047-a7cb-c97e3cb35e0f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7df488fbbd-smcmr\" (UID: \"b7258f31-b6f8-4047-a7cb-c97e3cb35e0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"
Sep 17 17:41:56.108943 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.108868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.109073 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.109054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ftz\" (UniqueName: \"kubernetes.io/projected/83fb8aed-02e5-47c9-a9a3-e5091da539fe-kube-api-access-f2ftz\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.109132 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.109097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.109184 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.109130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-ca\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.109184 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.109160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-hub\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.109715 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.109692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fb8aed-02e5-47c9-a9a3-e5091da539fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.111228 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.111202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.111478 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.111424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7258f31-b6f8-4047-a7cb-c97e3cb35e0f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7df488fbbd-smcmr\" (UID: \"b7258f31-b6f8-4047-a7cb-c97e3cb35e0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"
Sep 17 17:41:56.111478 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.111464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-ca\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.111864 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.111846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.111998 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.111955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/83fb8aed-02e5-47c9-a9a3-e5091da539fe-hub\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.119940 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.119923    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.140561 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.140541    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-794f84759f-p2xmg"]
Sep 17 17:41:56.159164 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.159142    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ftz\" (UniqueName: \"kubernetes.io/projected/83fb8aed-02e5-47c9-a9a3-e5091da539fe-kube-api-access-f2ftz\") pod \"cluster-proxy-proxy-agent-84fd97895b-x94j4\" (UID: \"83fb8aed-02e5-47c9-a9a3-e5091da539fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.159243 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.159188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gwt5\" (UniqueName: \"kubernetes.io/projected/b7258f31-b6f8-4047-a7cb-c97e3cb35e0f-kube-api-access-7gwt5\") pod \"managed-serviceaccount-addon-agent-7df488fbbd-smcmr\" (UID: \"b7258f31-b6f8-4047-a7cb-c97e3cb35e0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"
Sep 17 17:41:56.209967 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.209928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c51397f-ff44-4b47-9a0e-2e9df383c07b-image-registry-private-configuration\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.210094 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.209994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c51397f-ff44-4b47-9a0e-2e9df383c07b-trusted-ca\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.210094 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.210034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-bound-sa-token\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.210094 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.210062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-registry-tls\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.210094 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.210089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-789qm\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-kube-api-access-789qm\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.210295 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.210142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c51397f-ff44-4b47-9a0e-2e9df383c07b-ca-trust-extracted\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.210295 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.210157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c51397f-ff44-4b47-9a0e-2e9df383c07b-installation-pull-secrets\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.210295 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.210186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c51397f-ff44-4b47-9a0e-2e9df383c07b-registry-certificates\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.244378 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.244351    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"
Sep 17 17:41:56.251516 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.251497    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"
Sep 17 17:41:56.311018 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.310929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-registry-tls\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.311018 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.310980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789qm\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-kube-api-access-789qm\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.311902 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.311306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c51397f-ff44-4b47-9a0e-2e9df383c07b-ca-trust-extracted\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.311902 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.311343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c51397f-ff44-4b47-9a0e-2e9df383c07b-installation-pull-secrets\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.311902 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.311407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c51397f-ff44-4b47-9a0e-2e9df383c07b-registry-certificates\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.311902 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.311503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c51397f-ff44-4b47-9a0e-2e9df383c07b-image-registry-private-configuration\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.311902 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.311527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c51397f-ff44-4b47-9a0e-2e9df383c07b-trusted-ca\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.311902 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.311551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-bound-sa-token\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.311902 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.311779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c51397f-ff44-4b47-9a0e-2e9df383c07b-ca-trust-extracted\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.313166 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.313115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c51397f-ff44-4b47-9a0e-2e9df383c07b-registry-certificates\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.313262 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.313163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c51397f-ff44-4b47-9a0e-2e9df383c07b-trusted-ca\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.315809 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.315760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c51397f-ff44-4b47-9a0e-2e9df383c07b-image-registry-private-configuration\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.318547 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.318505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c51397f-ff44-4b47-9a0e-2e9df383c07b-installation-pull-secrets\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.319166 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.319123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-registry-tls\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.328072 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.328046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-789qm\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-kube-api-access-789qm\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.329706 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.329663    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c51397f-ff44-4b47-9a0e-2e9df383c07b-bound-sa-token\") pod \"image-registry-794f84759f-p2xmg\" (UID: \"2c51397f-ff44-4b47-9a0e-2e9df383c07b\") " pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.444456 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.444357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:56.542484 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:56.542449    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7258f31_b6f8_4047_a7cb_c97e3cb35e0f.slice/crio-4abf2d95d81eee59e18781b70767cf0685885463fc784b43bfea27ae6c01be4a WatchSource:0}: Error finding container 4abf2d95d81eee59e18781b70767cf0685885463fc784b43bfea27ae6c01be4a: Status 404 returned error can't find the container with id 4abf2d95d81eee59e18781b70767cf0685885463fc784b43bfea27ae6c01be4a
Sep 17 17:41:56.544075 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.544055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr"]
Sep 17 17:41:56.569994 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.569965    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4"]
Sep 17 17:41:56.573511 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:41:56.573488    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83fb8aed_02e5_47c9_a9a3_e5091da539fe.slice/crio-cfb25c2a2a2091379c7d7018ff0421927e154090cfe41bf0be0c5cb0377ebcef WatchSource:0}: Error finding container cfb25c2a2a2091379c7d7018ff0421927e154090cfe41bf0be0c5cb0377ebcef: Status 404 returned error can't find the container with id cfb25c2a2a2091379c7d7018ff0421927e154090cfe41bf0be0c5cb0377ebcef
Sep 17 17:41:56.629031 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.629000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4" event={"ID":"83fb8aed-02e5-47c9-a9a3-e5091da539fe","Type":"ContainerStarted","Data":"cfb25c2a2a2091379c7d7018ff0421927e154090cfe41bf0be0c5cb0377ebcef"}
Sep 17 17:41:56.630100 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.630076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr" event={"ID":"b7258f31-b6f8-4047-a7cb-c97e3cb35e0f","Type":"ContainerStarted","Data":"4abf2d95d81eee59e18781b70767cf0685885463fc784b43bfea27ae6c01be4a"}
Sep 17 17:41:56.753662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:56.753599    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-794f84759f-p2xmg"]
Sep 17 17:41:57.637305 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:57.636871    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794f84759f-p2xmg" event={"ID":"2c51397f-ff44-4b47-9a0e-2e9df383c07b","Type":"ContainerStarted","Data":"c83fad539d08be05db6da6c158e4d49003bc3cc61c65754386e3802922f8d5b4"}
Sep 17 17:41:57.637305 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:57.636931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794f84759f-p2xmg" event={"ID":"2c51397f-ff44-4b47-9a0e-2e9df383c07b","Type":"ContainerStarted","Data":"a96bea877ca5491ae3c98352285d097c32c316b4304b3df13f3ffe91ff95741c"}
Sep 17 17:41:57.637305 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:57.637257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:41:57.706894 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:57.706165    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-794f84759f-p2xmg" podStartSLOduration=1.706144805 podStartE2EDuration="1.706144805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:57.704864908 +0000 UTC m=+60.540634131" watchObservedRunningTime="2026-09-17 17:41:57.706144805 +0000 UTC m=+60.541914028"
Sep 17 17:41:58.535829 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:41:58.535643    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jk425"
Sep 17 17:42:01.615938 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:01.615905    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6nx7f"
Sep 17 17:42:01.656009 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:01.655978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4" event={"ID":"83fb8aed-02e5-47c9-a9a3-e5091da539fe","Type":"ContainerStarted","Data":"3618bcff6410ab3f541e16a051793b8b0e4dafb15afcf846451409e299c40f26"}
Sep 17 17:42:01.656180 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:01.656016    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4" event={"ID":"83fb8aed-02e5-47c9-a9a3-e5091da539fe","Type":"ContainerStarted","Data":"fb6da9a33053e50359dffc7693892a4209426d69aad26d11101f5fa30e511e4f"}
Sep 17 17:42:01.656180 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:01.656036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4" event={"ID":"83fb8aed-02e5-47c9-a9a3-e5091da539fe","Type":"ContainerStarted","Data":"b888205818c5b6418047668cb9f97198f44548c43f3f297a51dc7341028ad925"}
Sep 17 17:42:01.657561 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:01.657528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr" event={"ID":"b7258f31-b6f8-4047-a7cb-c97e3cb35e0f","Type":"ContainerStarted","Data":"95d25ae077dca93310c2d35015091f0143960c6e6b2a76697fb9ba81dd002bc7"}
Sep 17 17:42:01.676016 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:01.675961    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84fd97895b-x94j4" podStartSLOduration=2.644758204 podStartE2EDuration="6.675941667s" totalImagesPullingTime="4.031183463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:01.674469918 +0000 UTC m=+64.510239141" watchObservedRunningTime="2026-09-17 17:42:01.675941667 +0000 UTC m=+64.511710891"
Sep 17 17:42:01.689029 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:01.688991    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7df488fbbd-smcmr" podStartSLOduration=2.634615764 podStartE2EDuration="6.688972896s" totalImagesPullingTime="4.054357132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:01.68806856 +0000 UTC m=+64.523837820" watchObservedRunningTime="2026-09-17 17:42:01.688972896 +0000 UTC m=+64.524742118"
Sep 17 17:42:04.984223 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:04.984177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:42:04.986562 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:04.986540    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbc275f3-5fe6-40d8-b267-5cc0c7e7e143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fj7nq\" (UID: \"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143\") " pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:42:05.084808 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.084775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:42:05.084945 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.084821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:42:05.088236 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.088219    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:42:05.088463 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.088450    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:42:05.097195 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.097165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45898ed0-e311-492a-ab02-7851051c85a9-original-pull-secret\") pod \"global-pull-secret-syncer-v47zx\" (UID: \"45898ed0-e311-492a-ab02-7851051c85a9\") " pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:42:05.097320 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.097308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e9329e-dfb9-4ca8-9807-995754f7f505-metrics-certs\") pod \"network-metrics-daemon-x9dql\" (UID: \"06e9329e-dfb9-4ca8-9807-995754f7f505\") " pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:42:05.185812 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.185789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:42:05.188081 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.188060    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:42:05.192148 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.192132    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zrqgb\""
Sep 17 17:42:05.198238 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.198222    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:42:05.199981 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.199966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fj7nq"
Sep 17 17:42:05.208895 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.208851    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rpg\" (UniqueName: \"kubernetes.io/projected/d1b48b76-797b-4760-9783-3a1b06ea75b7-kube-api-access-27rpg\") pod \"network-check-target-5gtth\" (UID: \"d1b48b76-797b-4760-9783-3a1b06ea75b7\") " pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:42:05.253699 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.253673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v47zx"
Sep 17 17:42:05.262866 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.262049    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ccxk9\""
Sep 17 17:42:05.265701 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.265679    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7j64\""
Sep 17 17:42:05.270794 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.270774    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:42:05.274568 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.274551    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9dql"
Sep 17 17:42:05.478288 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.478260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fj7nq"]
Sep 17 17:42:05.481374 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:42:05.481348    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc275f3_5fe6_40d8_b267_5cc0c7e7e143.slice/crio-9d935083f11dd397e28615250d000dc5263485c88198c39a5a9e747523e65337 WatchSource:0}: Error finding container 9d935083f11dd397e28615250d000dc5263485c88198c39a5a9e747523e65337: Status 404 returned error can't find the container with id 9d935083f11dd397e28615250d000dc5263485c88198c39a5a9e747523e65337
Sep 17 17:42:05.540364 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.540342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v47zx"]
Sep 17 17:42:05.544171 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:42:05.544143    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45898ed0_e311_492a_ab02_7851051c85a9.slice/crio-1d83e82362cfd69f040083997d5b15062bd508b14855f984e85a3d3bba832726 WatchSource:0}: Error finding container 1d83e82362cfd69f040083997d5b15062bd508b14855f984e85a3d3bba832726: Status 404 returned error can't find the container with id 1d83e82362cfd69f040083997d5b15062bd508b14855f984e85a3d3bba832726
Sep 17 17:42:05.569575 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.569557    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gtth"]
Sep 17 17:42:05.571490 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:42:05.571467    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b48b76_797b_4760_9783_3a1b06ea75b7.slice/crio-0150676a13156796ffe7493eb7f01184abec5498514dccfe457a2c9b54216e4b WatchSource:0}: Error finding container 0150676a13156796ffe7493eb7f01184abec5498514dccfe457a2c9b54216e4b: Status 404 returned error can't find the container with id 0150676a13156796ffe7493eb7f01184abec5498514dccfe457a2c9b54216e4b
Sep 17 17:42:05.575349 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.575303    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x9dql"]
Sep 17 17:42:05.578566 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:42:05.578546    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e9329e_dfb9_4ca8_9807_995754f7f505.slice/crio-8ae9795200181cec56ab5e1490558868e1adfad3fa34c26fbda2e651c44bf8dd WatchSource:0}: Error finding container 8ae9795200181cec56ab5e1490558868e1adfad3fa34c26fbda2e651c44bf8dd: Status 404 returned error can't find the container with id 8ae9795200181cec56ab5e1490558868e1adfad3fa34c26fbda2e651c44bf8dd
Sep 17 17:42:05.673385 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.673354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9dql" event={"ID":"06e9329e-dfb9-4ca8-9807-995754f7f505","Type":"ContainerStarted","Data":"8ae9795200181cec56ab5e1490558868e1adfad3fa34c26fbda2e651c44bf8dd"}
Sep 17 17:42:05.674528 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.674503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gtth" event={"ID":"d1b48b76-797b-4760-9783-3a1b06ea75b7","Type":"ContainerStarted","Data":"0150676a13156796ffe7493eb7f01184abec5498514dccfe457a2c9b54216e4b"}
Sep 17 17:42:05.675526 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.675505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v47zx" event={"ID":"45898ed0-e311-492a-ab02-7851051c85a9","Type":"ContainerStarted","Data":"1d83e82362cfd69f040083997d5b15062bd508b14855f984e85a3d3bba832726"}
Sep 17 17:42:05.676914 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.676893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fj7nq" event={"ID":"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143","Type":"ContainerStarted","Data":"43794d84c6b466ce3d8336df6d47aa0096f61714f66c3e92c66a95ea4583d78d"}
Sep 17 17:42:05.676992 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.676931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fj7nq" event={"ID":"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143","Type":"ContainerStarted","Data":"9d935083f11dd397e28615250d000dc5263485c88198c39a5a9e747523e65337"}
Sep 17 17:42:05.984508 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.984461    2565 patch_prober.go:28] interesting pod/image-registry-645b995f97-6746s container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:42:05.984954 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:05.984532    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-645b995f97-6746s" podUID="1b093799-bfd9-46a6-ae5f-181eab889604" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:42:06.691337 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:06.691303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fj7nq" event={"ID":"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143","Type":"ContainerStarted","Data":"2d32fe86741b4117192e6f1497e383fbfc5dfa43661c0211988cf4e26f3cff6d"}
Sep 17 17:42:07.698057 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:07.698021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9dql" event={"ID":"06e9329e-dfb9-4ca8-9807-995754f7f505","Type":"ContainerStarted","Data":"af23a3fe3896ef3d86a1d952dc1cc1d7ab3d3c7a17b89da4d55913f7abc5e64c"}
Sep 17 17:42:07.698581 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:07.698077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9dql" event={"ID":"06e9329e-dfb9-4ca8-9807-995754f7f505","Type":"ContainerStarted","Data":"7d6c468bef64dd28a26f98a90c7019e5b2b211eec9f334b7700eab19381944b3"}
Sep 17 17:42:07.748311 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:07.747893    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x9dql" podStartSLOduration=67.677408819 podStartE2EDuration="1m8.74786791s" totalImagesPullingTime="1.070459091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:07.747058065 +0000 UTC m=+70.582827288" watchObservedRunningTime="2026-09-17 17:42:07.74786791 +0000 UTC m=+70.583637134"
Sep 17 17:42:10.710130 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:10.710094    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gtth" event={"ID":"d1b48b76-797b-4760-9783-3a1b06ea75b7","Type":"ContainerStarted","Data":"f8d5f51bdee34e7f6b9e4edaec81a9ed67c2adf802eb7cac01452153f6a7b541"}
Sep 17 17:42:10.710600 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:10.710191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:42:10.711647 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:10.711620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v47zx" event={"ID":"45898ed0-e311-492a-ab02-7851051c85a9","Type":"ContainerStarted","Data":"2f6864469b46149d2e298a2df9078d32528edf8f04e8ba91af406a266bb6ba66"}
Sep 17 17:42:10.713661 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:10.713641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fj7nq" event={"ID":"fbc275f3-5fe6-40d8-b267-5cc0c7e7e143","Type":"ContainerStarted","Data":"acef8f71fec8aeb300d8e0e62e86c1f711dbe65da4dd0908b6f894c5a08fe7cc"}
Sep 17 17:42:10.736409 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:10.736353    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5gtth" podStartSLOduration=67.065292282 podStartE2EDuration="1m11.73633825s" totalImagesPullingTime="4.671045968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:10.735676111 +0000 UTC m=+73.571445333" watchObservedRunningTime="2026-09-17 17:42:10.73633825 +0000 UTC m=+73.572107477"
Sep 17 17:42:10.758131 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:10.758090    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fj7nq" podStartSLOduration=34.135358292 podStartE2EDuration="38.758075669s" totalImagesPullingTime="4.622717377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:10.757237623 +0000 UTC m=+73.593006846" watchObservedRunningTime="2026-09-17 17:42:10.758075669 +0000 UTC m=+73.593844910"
Sep 17 17:42:15.227219 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.227154    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v47zx" podStartSLOduration=71.524603113 podStartE2EDuration="1m16.227108758s" totalImagesPullingTime="4.702505645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:10.795684147 +0000 UTC m=+73.631453369" watchObservedRunningTime="2026-09-17 17:42:15.227108758 +0000 UTC m=+78.062878044"
Sep 17 17:42:15.228157 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.228136    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zf6cw"]
Sep 17 17:42:15.230570 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.230552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.234992 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.234971    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:42:15.235103 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.235050    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:42:15.235390 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.235372    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qffnd\""
Sep 17 17:42:15.235648 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.235633    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:42:15.235733 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.235676    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:42:15.236033 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.236020    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:42:15.282819 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.282800    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:42:15.366975 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.366948    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tkj\" (UniqueName: \"kubernetes.io/projected/8acb1be4-db48-44cc-883c-1ed598a3cd87-kube-api-access-w8tkj\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.367089 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.366977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-tls\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.367089 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.366998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-root\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.367089 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.367030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8acb1be4-db48-44cc-883c-1ed598a3cd87-metrics-client-ca\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.367189 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.367093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.367189 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.367126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-sys\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.367189 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.367164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-wtmp\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.367276 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.367189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.367276 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.367205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-textfile\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468282 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-root\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8acb1be4-db48-44cc-883c-1ed598a3cd87-metrics-client-ca\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-sys\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-wtmp\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468368    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-root\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-sys\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-textfile\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468887 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-wtmp\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468887 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tkj\" (UniqueName: \"kubernetes.io/projected/8acb1be4-db48-44cc-883c-1ed598a3cd87-kube-api-access-w8tkj\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468887 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-tls\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.468887 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-textfile\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.469082 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.468953    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.469082 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.469011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8acb1be4-db48-44cc-883c-1ed598a3cd87-metrics-client-ca\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.470816 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.470796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.471115 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.471100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8acb1be4-db48-44cc-883c-1ed598a3cd87-node-exporter-tls\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.482102 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.482046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tkj\" (UniqueName: \"kubernetes.io/projected/8acb1be4-db48-44cc-883c-1ed598a3cd87-kube-api-access-w8tkj\") pod \"node-exporter-zf6cw\" (UID: \"8acb1be4-db48-44cc-883c-1ed598a3cd87\") " pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.539177 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.539145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zf6cw"
Sep 17 17:42:15.549670 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:42:15.549649    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acb1be4_db48_44cc_883c_1ed598a3cd87.slice/crio-ab0648eb889cf4dffe3f449e361a4397f50cd4a4dafb70912de17a7cc85d4fdd WatchSource:0}: Error finding container ab0648eb889cf4dffe3f449e361a4397f50cd4a4dafb70912de17a7cc85d4fdd: Status 404 returned error can't find the container with id ab0648eb889cf4dffe3f449e361a4397f50cd4a4dafb70912de17a7cc85d4fdd
Sep 17 17:42:15.731416 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.731362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf6cw" event={"ID":"8acb1be4-db48-44cc-883c-1ed598a3cd87","Type":"ContainerStarted","Data":"ab0648eb889cf4dffe3f449e361a4397f50cd4a4dafb70912de17a7cc85d4fdd"}
Sep 17 17:42:15.982443 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:15.982410    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:42:16.736801 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:16.736766    2565 generic.go:356] "Generic (PLEG): container finished" podID="8acb1be4-db48-44cc-883c-1ed598a3cd87" containerID="f4afac765698037988e9786ef231490cab88223ab47e2fe23fda2ca28f6d1b25" exitCode=0
Sep 17 17:42:16.737209 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:16.736825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf6cw" event={"ID":"8acb1be4-db48-44cc-883c-1ed598a3cd87","Type":"ContainerDied","Data":"f4afac765698037988e9786ef231490cab88223ab47e2fe23fda2ca28f6d1b25"}
Sep 17 17:42:17.745849 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:17.745809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf6cw" event={"ID":"8acb1be4-db48-44cc-883c-1ed598a3cd87","Type":"ContainerStarted","Data":"6b9bdaf68dc7c91cf71bfe5762bffa49da68b1f4664d6ba38b3d2e6a39147086"}
Sep 17 17:42:17.746226 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:17.745901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf6cw" event={"ID":"8acb1be4-db48-44cc-883c-1ed598a3cd87","Type":"ContainerStarted","Data":"6543d2d6ac5e59db12fc95fd71bf9e70a237257d484006227e7b97c4f0dd2e5f"}
Sep 17 17:42:18.644920 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:18.644889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-794f84759f-p2xmg"
Sep 17 17:42:18.685518 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:18.685468    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zf6cw" podStartSLOduration=2.643411147 podStartE2EDuration="3.685451929s" totalImagesPullingTime="979.109113ms" totalInitContainerRuntime="62.931669ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:17.786371643 +0000 UTC m=+80.622140866" watchObservedRunningTime="2026-09-17 17:42:18.685451929 +0000 UTC m=+81.521221152"
Sep 17 17:42:20.994552 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:20.994513    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-645b995f97-6746s" podUID="1b093799-bfd9-46a6-ae5f-181eab889604" containerName="registry" containerID="cri-o://3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc" gracePeriod=30
Sep 17 17:42:21.240948 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.240926    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:42:21.418080 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") pod \"1b093799-bfd9-46a6-ae5f-181eab889604\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") "
Sep 17 17:42:21.418212 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-installation-pull-secrets\") pod \"1b093799-bfd9-46a6-ae5f-181eab889604\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") "
Sep 17 17:42:21.418212 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418118    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-trusted-ca\") pod \"1b093799-bfd9-46a6-ae5f-181eab889604\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") "
Sep 17 17:42:21.418212 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418140    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czdw4\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-kube-api-access-czdw4\") pod \"1b093799-bfd9-46a6-ae5f-181eab889604\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") "
Sep 17 17:42:21.418369 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418288    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b093799-bfd9-46a6-ae5f-181eab889604-ca-trust-extracted\") pod \"1b093799-bfd9-46a6-ae5f-181eab889604\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") "
Sep 17 17:42:21.418369 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418332    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-bound-sa-token\") pod \"1b093799-bfd9-46a6-ae5f-181eab889604\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") "
Sep 17 17:42:21.418513 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-image-registry-private-configuration\") pod \"1b093799-bfd9-46a6-ae5f-181eab889604\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") "
Sep 17 17:42:21.418513 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-registry-certificates\") pod \"1b093799-bfd9-46a6-ae5f-181eab889604\" (UID: \"1b093799-bfd9-46a6-ae5f-181eab889604\") "
Sep 17 17:42:21.418611 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1b093799-bfd9-46a6-ae5f-181eab889604" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:21.418696 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418678    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-trusted-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:42:21.418947 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.418914    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1b093799-bfd9-46a6-ae5f-181eab889604" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:21.421078 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.421049    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1b093799-bfd9-46a6-ae5f-181eab889604" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:21.421078 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.421060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1b093799-bfd9-46a6-ae5f-181eab889604" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:21.421226 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.421141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1b093799-bfd9-46a6-ae5f-181eab889604" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:21.421226 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.421178    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1b093799-bfd9-46a6-ae5f-181eab889604" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:21.421226 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.421200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-kube-api-access-czdw4" (OuterVolumeSpecName: "kube-api-access-czdw4") pod "1b093799-bfd9-46a6-ae5f-181eab889604" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604"). InnerVolumeSpecName "kube-api-access-czdw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:21.429155 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.429127    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b093799-bfd9-46a6-ae5f-181eab889604-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1b093799-bfd9-46a6-ae5f-181eab889604" (UID: "1b093799-bfd9-46a6-ae5f-181eab889604"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:21.519455 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.519421    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-image-registry-private-configuration\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:42:21.519455 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.519453    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b093799-bfd9-46a6-ae5f-181eab889604-registry-certificates\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:42:21.519455 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.519465    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-registry-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:42:21.519609 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.519474    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b093799-bfd9-46a6-ae5f-181eab889604-installation-pull-secrets\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:42:21.519609 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.519482    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdw4\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-kube-api-access-czdw4\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:42:21.519609 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.519491    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b093799-bfd9-46a6-ae5f-181eab889604-ca-trust-extracted\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:42:21.519609 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.519500    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b093799-bfd9-46a6-ae5f-181eab889604-bound-sa-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:42:21.760841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.760746    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b093799-bfd9-46a6-ae5f-181eab889604" containerID="3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc" exitCode=0
Sep 17 17:42:21.760841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.760809    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645b995f97-6746s"
Sep 17 17:42:21.760841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.760831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645b995f97-6746s" event={"ID":"1b093799-bfd9-46a6-ae5f-181eab889604","Type":"ContainerDied","Data":"3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc"}
Sep 17 17:42:21.761115 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.760884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645b995f97-6746s" event={"ID":"1b093799-bfd9-46a6-ae5f-181eab889604","Type":"ContainerDied","Data":"88bfa448fa81da518ac20e26c8d68612ef824546429157bfd2fce038f9a1b78f"}
Sep 17 17:42:21.761115 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.760905    2565 scope.go:117] "RemoveContainer" containerID="3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc"
Sep 17 17:42:21.769264 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.769249    2565 scope.go:117] "RemoveContainer" containerID="3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc"
Sep 17 17:42:21.769569 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:42:21.769548    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc\": container with ID starting with 3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc not found: ID does not exist" containerID="3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc"
Sep 17 17:42:21.769652 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.769575    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc"} err="failed to get container status \"3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc\": rpc error: code = NotFound desc = could not find container \"3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc\": container with ID starting with 3eed5534f9d79dd17e89ad8032a858cd58297afcc616c0dedc076e168b187cbc not found: ID does not exist"
Sep 17 17:42:21.810035 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.810008    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-645b995f97-6746s"]
Sep 17 17:42:21.816499 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:21.816478    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-645b995f97-6746s"]
Sep 17 17:42:23.347683 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:23.347651    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b093799-bfd9-46a6-ae5f-181eab889604" path="/var/lib/kubelet/pods/1b093799-bfd9-46a6-ae5f-181eab889604/volumes"
Sep 17 17:42:41.719519 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:42:41.719489    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5gtth"
Sep 17 17:45:59.255094 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:45:59.255069    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:45:59.261998 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:45:59.261974    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:45:59.264980 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:45:59.264960    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:47:32.127020 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.126951    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jt4hf"]
Sep 17 17:47:32.127651 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.127295    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b093799-bfd9-46a6-ae5f-181eab889604" containerName="registry"
Sep 17 17:47:32.127651 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.127314    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b093799-bfd9-46a6-ae5f-181eab889604" containerName="registry"
Sep 17 17:47:32.127651 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.127411    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b093799-bfd9-46a6-ae5f-181eab889604" containerName="registry"
Sep 17 17:47:32.130016 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.129996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jt4hf"
Sep 17 17:47:32.132185 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.132165    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 17:47:32.132636 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.132610    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6kx9n\""
Sep 17 17:47:32.132706 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.132646    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 17:47:32.140889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.140871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jt4hf"]
Sep 17 17:47:32.206696 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.206665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtth2\" (UniqueName: \"kubernetes.io/projected/2d10411b-b712-4fd2-942f-288f79663309-kube-api-access-mtth2\") pod \"cert-manager-55b4d954d5-jt4hf\" (UID: \"2d10411b-b712-4fd2-942f-288f79663309\") " pod="cert-manager/cert-manager-55b4d954d5-jt4hf"
Sep 17 17:47:32.206852 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.206707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d10411b-b712-4fd2-942f-288f79663309-bound-sa-token\") pod \"cert-manager-55b4d954d5-jt4hf\" (UID: \"2d10411b-b712-4fd2-942f-288f79663309\") " pod="cert-manager/cert-manager-55b4d954d5-jt4hf"
Sep 17 17:47:32.307925 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.307897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtth2\" (UniqueName: \"kubernetes.io/projected/2d10411b-b712-4fd2-942f-288f79663309-kube-api-access-mtth2\") pod \"cert-manager-55b4d954d5-jt4hf\" (UID: \"2d10411b-b712-4fd2-942f-288f79663309\") " pod="cert-manager/cert-manager-55b4d954d5-jt4hf"
Sep 17 17:47:32.308037 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.307934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d10411b-b712-4fd2-942f-288f79663309-bound-sa-token\") pod \"cert-manager-55b4d954d5-jt4hf\" (UID: \"2d10411b-b712-4fd2-942f-288f79663309\") " pod="cert-manager/cert-manager-55b4d954d5-jt4hf"
Sep 17 17:47:32.316437 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.316416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d10411b-b712-4fd2-942f-288f79663309-bound-sa-token\") pod \"cert-manager-55b4d954d5-jt4hf\" (UID: \"2d10411b-b712-4fd2-942f-288f79663309\") " pod="cert-manager/cert-manager-55b4d954d5-jt4hf"
Sep 17 17:47:32.316559 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.316471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtth2\" (UniqueName: \"kubernetes.io/projected/2d10411b-b712-4fd2-942f-288f79663309-kube-api-access-mtth2\") pod \"cert-manager-55b4d954d5-jt4hf\" (UID: \"2d10411b-b712-4fd2-942f-288f79663309\") " pod="cert-manager/cert-manager-55b4d954d5-jt4hf"
Sep 17 17:47:32.439168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.439138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jt4hf"
Sep 17 17:47:32.707973 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.707897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jt4hf"]
Sep 17 17:47:32.710952 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:47:32.710920    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d10411b_b712_4fd2_942f_288f79663309.slice/crio-68fc61ffa6e91ced9d8d37d0cbd6bcd28cbf9507c4295e0a2a1efc10388c2abd WatchSource:0}: Error finding container 68fc61ffa6e91ced9d8d37d0cbd6bcd28cbf9507c4295e0a2a1efc10388c2abd: Status 404 returned error can't find the container with id 68fc61ffa6e91ced9d8d37d0cbd6bcd28cbf9507c4295e0a2a1efc10388c2abd
Sep 17 17:47:32.713273 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.713258    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:47:32.771001 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:32.770972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jt4hf" event={"ID":"2d10411b-b712-4fd2-942f-288f79663309","Type":"ContainerStarted","Data":"68fc61ffa6e91ced9d8d37d0cbd6bcd28cbf9507c4295e0a2a1efc10388c2abd"}
Sep 17 17:47:36.787577 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:36.787546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jt4hf" event={"ID":"2d10411b-b712-4fd2-942f-288f79663309","Type":"ContainerStarted","Data":"9acf076acf6679fd5b2519e36706456e9c99970f98935447e0f46187bab8e723"}
Sep 17 17:47:36.805151 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:47:36.805100    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jt4hf" podStartSLOduration=1.790460952 podStartE2EDuration="4.805084623s" totalImagesPullingTime="3.014623671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:36.804355266 +0000 UTC m=+399.640124488" watchObservedRunningTime="2026-09-17 17:47:36.805084623 +0000 UTC m=+399.640853862"
Sep 17 17:48:01.807862 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.807832    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"]
Sep 17 17:48:01.812504 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.812482    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.817042 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.817019    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:48:01.817188 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.817026    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-hk9zk\""
Sep 17 17:48:01.817285 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.817270    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:48:01.817963 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.817948    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:48:01.828458 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.828437    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"]
Sep 17 17:48:01.915765 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.915765 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.916008 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.916008 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.916008 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.916008 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.916008 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.916008 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:01.916008 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:01.915986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4lb\" (UniqueName: \"kubernetes.io/projected/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-kube-api-access-wf4lb\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.016929 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.016894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4lb\" (UniqueName: \"kubernetes.io/projected/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-kube-api-access-wf4lb\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017095 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.016958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017095 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.016981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017095 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017095 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017095 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017095 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017416 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017416 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017416 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.017820 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.017804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.019362 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.019342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.019620 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.019594    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.033659 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.033638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4lb\" (UniqueName: \"kubernetes.io/projected/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-kube-api-access-wf4lb\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.036889 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.036869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9ca3db-8b65-44b4-a00e-71ff7578fefb-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc\" (UID: \"fc9ca3db-8b65-44b4-a00e-71ff7578fefb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.124018 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.123951    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:02.410105 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.410079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"]
Sep 17 17:48:02.413177 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:48:02.413155    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9ca3db_8b65_44b4_a00e_71ff7578fefb.slice/crio-593f8e33d6aa17825d5738cdde66c6f1dfdb4396483de87273a8e763b795f858 WatchSource:0}: Error finding container 593f8e33d6aa17825d5738cdde66c6f1dfdb4396483de87273a8e763b795f858: Status 404 returned error can't find the container with id 593f8e33d6aa17825d5738cdde66c6f1dfdb4396483de87273a8e763b795f858
Sep 17 17:48:02.873361 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:02.873277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" event={"ID":"fc9ca3db-8b65-44b4-a00e-71ff7578fefb","Type":"ContainerStarted","Data":"593f8e33d6aa17825d5738cdde66c6f1dfdb4396483de87273a8e763b795f858"}
Sep 17 17:48:05.355944 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:05.355910    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:48:05.356203 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:05.355979    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:48:05.356203 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:05.356006    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:48:05.886508 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:05.886473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" event={"ID":"fc9ca3db-8b65-44b4-a00e-71ff7578fefb","Type":"ContainerStarted","Data":"9aec79c2d3b90c5b6ac79992b681eff043fc179d87bc39525edfb4b247ffc35e"}
Sep 17 17:48:05.910766 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:05.910704    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" podStartSLOduration=1.970126696 podStartE2EDuration="4.910684502s" totalImagesPullingTime="2.940557806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:05.907736262 +0000 UTC m=+428.743505485" watchObservedRunningTime="2026-09-17 17:48:05.910684502 +0000 UTC m=+428.746453750"
Sep 17 17:48:06.124084 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:06.124054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:06.128658 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:06.128634    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:06.890243 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:06.890217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:06.891052 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:06.891037    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:48:12.189671 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.189594    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"]
Sep 17 17:48:12.192827 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.192808    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.196547 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.196526    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 17:48:12.196806 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.196783    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 17:48:12.197122 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.197107    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8v7sm\""
Sep 17 17:48:12.197163 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.197121    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 17:48:12.201440 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.201423    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:48:12.201578 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.201563    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 17:48:12.221308 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.221286    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"]
Sep 17 17:48:12.299806 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.299781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzgh\" (UniqueName: \"kubernetes.io/projected/33497543-6e0c-4b2c-96e6-6690af1c6a82-kube-api-access-lzzgh\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.299954 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.299815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/33497543-6e0c-4b2c-96e6-6690af1c6a82-manager-config\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.299954 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.299851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/33497543-6e0c-4b2c-96e6-6690af1c6a82-metrics-cert\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.299954 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.299879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33497543-6e0c-4b2c-96e6-6690af1c6a82-cert\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.400925 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.400892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzgh\" (UniqueName: \"kubernetes.io/projected/33497543-6e0c-4b2c-96e6-6690af1c6a82-kube-api-access-lzzgh\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.400925 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.400925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/33497543-6e0c-4b2c-96e6-6690af1c6a82-manager-config\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.401142 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.400951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/33497543-6e0c-4b2c-96e6-6690af1c6a82-metrics-cert\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.401211 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.401190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33497543-6e0c-4b2c-96e6-6690af1c6a82-cert\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.401671 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.401645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/33497543-6e0c-4b2c-96e6-6690af1c6a82-manager-config\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.403353 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.403330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/33497543-6e0c-4b2c-96e6-6690af1c6a82-metrics-cert\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.403483 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.403448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33497543-6e0c-4b2c-96e6-6690af1c6a82-cert\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.410790 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.410764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzgh\" (UniqueName: \"kubernetes.io/projected/33497543-6e0c-4b2c-96e6-6690af1c6a82-kube-api-access-lzzgh\") pod \"lws-controller-manager-68dc9b9bc-s7hzh\" (UID: \"33497543-6e0c-4b2c-96e6-6690af1c6a82\") " pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.501132 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.501053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:12.770439 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.770344    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"]
Sep 17 17:48:12.773082 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:48:12.773057    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33497543_6e0c_4b2c_96e6_6690af1c6a82.slice/crio-5087d81a8d6a5f89e2b4e2fc0ae27908b6f42b5411bd2b41957277b4742091ae WatchSource:0}: Error finding container 5087d81a8d6a5f89e2b4e2fc0ae27908b6f42b5411bd2b41957277b4742091ae: Status 404 returned error can't find the container with id 5087d81a8d6a5f89e2b4e2fc0ae27908b6f42b5411bd2b41957277b4742091ae
Sep 17 17:48:12.912496 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:12.912457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh" event={"ID":"33497543-6e0c-4b2c-96e6-6690af1c6a82","Type":"ContainerStarted","Data":"5087d81a8d6a5f89e2b4e2fc0ae27908b6f42b5411bd2b41957277b4742091ae"}
Sep 17 17:48:14.921432 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:14.921386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh" event={"ID":"33497543-6e0c-4b2c-96e6-6690af1c6a82","Type":"ContainerStarted","Data":"a83c64d352bcbec81383682f52e5cede0f933f5f1b56db20f88ccadb3d81cc63"}
Sep 17 17:48:15.924862 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:15.924833    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:15.943363 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:15.943310    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh" podStartSLOduration=1.8572680099999999 podStartE2EDuration="3.943296399s" totalImagesPullingTime="2.086028389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:15.94313812 +0000 UTC m=+438.778907353" watchObservedRunningTime="2026-09-17 17:48:15.943296399 +0000 UTC m=+438.779065637"
Sep 17 17:48:26.930783 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:26.930754    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68dc9b9bc-s7hzh"
Sep 17 17:48:40.082020 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.081985    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"]
Sep 17 17:48:40.089625 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.089607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.092538 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.092513    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 17:48:40.092665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.092553    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 17:48:40.092665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.092609    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2gct4\""
Sep 17 17:48:40.093186 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.093163    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 17:48:40.093297 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.093168    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 17:48:40.097367 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.097335    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"]
Sep 17 17:48:40.208567 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.208533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b709f81d-3c1a-4fe2-bdac-364bbab10227-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.208730 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.208637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsx2q\" (UniqueName: \"kubernetes.io/projected/b709f81d-3c1a-4fe2-bdac-364bbab10227-kube-api-access-zsx2q\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.208730 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.208663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b709f81d-3c1a-4fe2-bdac-364bbab10227-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.309252 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.309219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsx2q\" (UniqueName: \"kubernetes.io/projected/b709f81d-3c1a-4fe2-bdac-364bbab10227-kube-api-access-zsx2q\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.309252 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.309256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b709f81d-3c1a-4fe2-bdac-364bbab10227-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.309533 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.309305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b709f81d-3c1a-4fe2-bdac-364bbab10227-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.309533 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:48:40.309369    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 17 17:48:40.309533 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:48:40.309444    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b709f81d-3c1a-4fe2-bdac-364bbab10227-plugin-serving-cert podName:b709f81d-3c1a-4fe2-bdac-364bbab10227 nodeName:}" failed. No retries permitted until 2026-09-17 17:48:40.809429612 +0000 UTC m=+463.645198816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b709f81d-3c1a-4fe2-bdac-364bbab10227-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-g6gv9" (UID: "b709f81d-3c1a-4fe2-bdac-364bbab10227") : secret "plugin-serving-cert" not found
Sep 17 17:48:40.309978 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.309962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b709f81d-3c1a-4fe2-bdac-364bbab10227-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.320615 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.320586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsx2q\" (UniqueName: \"kubernetes.io/projected/b709f81d-3c1a-4fe2-bdac-364bbab10227-kube-api-access-zsx2q\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.814789 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.814744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b709f81d-3c1a-4fe2-bdac-364bbab10227-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.817181 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.817161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b709f81d-3c1a-4fe2-bdac-364bbab10227-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-g6gv9\" (UID: \"b709f81d-3c1a-4fe2-bdac-364bbab10227\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:40.999084 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:40.999052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"
Sep 17 17:48:41.268056 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:41.268024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9"]
Sep 17 17:48:41.270864 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:48:41.270831    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb709f81d_3c1a_4fe2_bdac_364bbab10227.slice/crio-165457cb34f117d851f3df5fc70d96288651aad6083e71100e397f04e8fb8f59 WatchSource:0}: Error finding container 165457cb34f117d851f3df5fc70d96288651aad6083e71100e397f04e8fb8f59: Status 404 returned error can't find the container with id 165457cb34f117d851f3df5fc70d96288651aad6083e71100e397f04e8fb8f59
Sep 17 17:48:42.020777 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:42.020711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9" event={"ID":"b709f81d-3c1a-4fe2-bdac-364bbab10227","Type":"ContainerStarted","Data":"165457cb34f117d851f3df5fc70d96288651aad6083e71100e397f04e8fb8f59"}
Sep 17 17:48:46.038386 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:46.038341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9" event={"ID":"b709f81d-3c1a-4fe2-bdac-364bbab10227","Type":"ContainerStarted","Data":"10b8e040372b5502b77d6978269d7c3a550f3752f3fe7741317a64b65e04cd4e"}
Sep 17 17:48:46.054971 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:48:46.054749    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g6gv9" podStartSLOduration=1.7064963880000001 podStartE2EDuration="6.054731102s" totalImagesPullingTime="4.348234714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:46.054539525 +0000 UTC m=+468.890308749" watchObservedRunningTime="2026-09-17 17:48:46.054731102 +0000 UTC m=+468.890500326"
Sep 17 17:49:20.782251 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:20.782216    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cxgh6"]
Sep 17 17:49:20.800000 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:20.799975    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cxgh6"]
Sep 17 17:49:20.800132 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:20.800062    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6"
Sep 17 17:49:20.802287 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:20.802261    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n7nvx\""
Sep 17 17:49:20.930298 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:20.930259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqf48\" (UniqueName: \"kubernetes.io/projected/46e62810-99f6-445b-b7a8-dfc2f957b51a-kube-api-access-kqf48\") pod \"authorino-7bcfd8f9cd-cxgh6\" (UID: \"46e62810-99f6-445b-b7a8-dfc2f957b51a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6"
Sep 17 17:49:21.031662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:21.031629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqf48\" (UniqueName: \"kubernetes.io/projected/46e62810-99f6-445b-b7a8-dfc2f957b51a-kube-api-access-kqf48\") pod \"authorino-7bcfd8f9cd-cxgh6\" (UID: \"46e62810-99f6-445b-b7a8-dfc2f957b51a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6"
Sep 17 17:49:21.040097 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:21.040037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqf48\" (UniqueName: \"kubernetes.io/projected/46e62810-99f6-445b-b7a8-dfc2f957b51a-kube-api-access-kqf48\") pod \"authorino-7bcfd8f9cd-cxgh6\" (UID: \"46e62810-99f6-445b-b7a8-dfc2f957b51a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6"
Sep 17 17:49:21.108564 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:21.108528    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6"
Sep 17 17:49:21.375413 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:21.375335    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cxgh6"]
Sep 17 17:49:21.379571 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:49:21.379546    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46e62810_99f6_445b_b7a8_dfc2f957b51a.slice/crio-3df11a157fe58d1df3d3e102d9113069e9087771543c848a3c2e6339ae338ca5 WatchSource:0}: Error finding container 3df11a157fe58d1df3d3e102d9113069e9087771543c848a3c2e6339ae338ca5: Status 404 returned error can't find the container with id 3df11a157fe58d1df3d3e102d9113069e9087771543c848a3c2e6339ae338ca5
Sep 17 17:49:22.171937 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:22.171898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6" event={"ID":"46e62810-99f6-445b-b7a8-dfc2f957b51a","Type":"ContainerStarted","Data":"3df11a157fe58d1df3d3e102d9113069e9087771543c848a3c2e6339ae338ca5"}
Sep 17 17:49:24.847387 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:24.847311    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cxgh6"]
Sep 17 17:49:25.185902 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:25.185873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6" event={"ID":"46e62810-99f6-445b-b7a8-dfc2f957b51a","Type":"ContainerStarted","Data":"cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d"}
Sep 17 17:49:25.201970 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:25.201928    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6" podStartSLOduration=2.1061854 podStartE2EDuration="5.201914492s" totalImagesPullingTime="3.095729092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:25.201744624 +0000 UTC m=+508.037513858" watchObservedRunningTime="2026-09-17 17:49:25.201914492 +0000 UTC m=+508.037683715"
Sep 17 17:49:26.189562 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:26.189522    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6" podUID="46e62810-99f6-445b-b7a8-dfc2f957b51a" containerName="authorino" containerID="cri-o://cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d" gracePeriod=30
Sep 17 17:49:26.422128 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:26.422099    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6"
Sep 17 17:49:26.577330 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:26.577239    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqf48\" (UniqueName: \"kubernetes.io/projected/46e62810-99f6-445b-b7a8-dfc2f957b51a-kube-api-access-kqf48\") pod \"46e62810-99f6-445b-b7a8-dfc2f957b51a\" (UID: \"46e62810-99f6-445b-b7a8-dfc2f957b51a\") "
Sep 17 17:49:26.579309 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:26.579282    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e62810-99f6-445b-b7a8-dfc2f957b51a-kube-api-access-kqf48" (OuterVolumeSpecName: "kube-api-access-kqf48") pod "46e62810-99f6-445b-b7a8-dfc2f957b51a" (UID: "46e62810-99f6-445b-b7a8-dfc2f957b51a"). InnerVolumeSpecName "kube-api-access-kqf48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:26.677755 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:26.677711    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqf48\" (UniqueName: \"kubernetes.io/projected/46e62810-99f6-445b-b7a8-dfc2f957b51a-kube-api-access-kqf48\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:49:27.194476 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.194443    2565 generic.go:356] "Generic (PLEG): container finished" podID="46e62810-99f6-445b-b7a8-dfc2f957b51a" containerID="cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d" exitCode=0
Sep 17 17:49:27.194896 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.194488    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6"
Sep 17 17:49:27.194896 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.194523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6" event={"ID":"46e62810-99f6-445b-b7a8-dfc2f957b51a","Type":"ContainerDied","Data":"cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d"}
Sep 17 17:49:27.194896 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.194566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cxgh6" event={"ID":"46e62810-99f6-445b-b7a8-dfc2f957b51a","Type":"ContainerDied","Data":"3df11a157fe58d1df3d3e102d9113069e9087771543c848a3c2e6339ae338ca5"}
Sep 17 17:49:27.194896 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.194588    2565 scope.go:117] "RemoveContainer" containerID="cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d"
Sep 17 17:49:27.203294 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.203189    2565 scope.go:117] "RemoveContainer" containerID="cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d"
Sep 17 17:49:27.203563 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:49:27.203543    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d\": container with ID starting with cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d not found: ID does not exist" containerID="cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d"
Sep 17 17:49:27.203616 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.203573    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d"} err="failed to get container status \"cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d\": rpc error: code = NotFound desc = could not find container \"cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d\": container with ID starting with cf0563f0781abbe66cf5e8c4bedca5d9c26102cfaf6ebe06d0be28288d3c4b0d not found: ID does not exist"
Sep 17 17:49:27.216436 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.216412    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cxgh6"]
Sep 17 17:49:27.219903 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.219883    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cxgh6"]
Sep 17 17:49:27.353461 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:27.353431    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46e62810-99f6-445b-b7a8-dfc2f957b51a" path="/var/lib/kubelet/pods/46e62810-99f6-445b-b7a8-dfc2f957b51a/volumes"
Sep 17 17:49:43.078622 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.078530    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-czwqb"]
Sep 17 17:49:43.078971 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.078836    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e62810-99f6-445b-b7a8-dfc2f957b51a" containerName="authorino"
Sep 17 17:49:43.078971 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.078850    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e62810-99f6-445b-b7a8-dfc2f957b51a" containerName="authorino"
Sep 17 17:49:43.078971 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.078904    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e62810-99f6-445b-b7a8-dfc2f957b51a" containerName="authorino"
Sep 17 17:49:43.081235 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.081221    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-czwqb"
Sep 17 17:49:43.084352 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.084329    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n7nvx\""
Sep 17 17:49:43.084492 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.084359    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 17:49:43.092611 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.092592    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-czwqb"]
Sep 17 17:49:43.182652 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.182620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vtf\" (UniqueName: \"kubernetes.io/projected/61af4be2-2835-47be-8d79-7e7cd3a26832-kube-api-access-k7vtf\") pod \"authorino-6d9898bd99-czwqb\" (UID: \"61af4be2-2835-47be-8d79-7e7cd3a26832\") " pod="kuadrant-system/authorino-6d9898bd99-czwqb"
Sep 17 17:49:43.182797 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.182688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61af4be2-2835-47be-8d79-7e7cd3a26832-tls-cert\") pod \"authorino-6d9898bd99-czwqb\" (UID: \"61af4be2-2835-47be-8d79-7e7cd3a26832\") " pod="kuadrant-system/authorino-6d9898bd99-czwqb"
Sep 17 17:49:43.283775 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.283742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vtf\" (UniqueName: \"kubernetes.io/projected/61af4be2-2835-47be-8d79-7e7cd3a26832-kube-api-access-k7vtf\") pod \"authorino-6d9898bd99-czwqb\" (UID: \"61af4be2-2835-47be-8d79-7e7cd3a26832\") " pod="kuadrant-system/authorino-6d9898bd99-czwqb"
Sep 17 17:49:43.283935 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.283796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61af4be2-2835-47be-8d79-7e7cd3a26832-tls-cert\") pod \"authorino-6d9898bd99-czwqb\" (UID: \"61af4be2-2835-47be-8d79-7e7cd3a26832\") " pod="kuadrant-system/authorino-6d9898bd99-czwqb"
Sep 17 17:49:43.286213 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.286186    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61af4be2-2835-47be-8d79-7e7cd3a26832-tls-cert\") pod \"authorino-6d9898bd99-czwqb\" (UID: \"61af4be2-2835-47be-8d79-7e7cd3a26832\") " pod="kuadrant-system/authorino-6d9898bd99-czwqb"
Sep 17 17:49:43.292083 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.292061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vtf\" (UniqueName: \"kubernetes.io/projected/61af4be2-2835-47be-8d79-7e7cd3a26832-kube-api-access-k7vtf\") pod \"authorino-6d9898bd99-czwqb\" (UID: \"61af4be2-2835-47be-8d79-7e7cd3a26832\") " pod="kuadrant-system/authorino-6d9898bd99-czwqb"
Sep 17 17:49:43.390023 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.389958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-czwqb"
Sep 17 17:49:43.660477 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:43.660446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-czwqb"]
Sep 17 17:49:43.663154 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:49:43.663125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61af4be2_2835_47be_8d79_7e7cd3a26832.slice/crio-10924ad1cc5303e829ea5ae289d5e48aa0460ef79d1b1c96ac740d9e0a5ae132 WatchSource:0}: Error finding container 10924ad1cc5303e829ea5ae289d5e48aa0460ef79d1b1c96ac740d9e0a5ae132: Status 404 returned error can't find the container with id 10924ad1cc5303e829ea5ae289d5e48aa0460ef79d1b1c96ac740d9e0a5ae132
Sep 17 17:49:44.259016 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:44.258986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-czwqb" event={"ID":"61af4be2-2835-47be-8d79-7e7cd3a26832","Type":"ContainerStarted","Data":"f5b64044464683e4220fd14df66162a349724d9dc807b58fd0f7c82fc4983185"}
Sep 17 17:49:44.259489 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:44.259027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-czwqb" event={"ID":"61af4be2-2835-47be-8d79-7e7cd3a26832","Type":"ContainerStarted","Data":"10924ad1cc5303e829ea5ae289d5e48aa0460ef79d1b1c96ac740d9e0a5ae132"}
Sep 17 17:49:44.275670 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:49:44.275613    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-czwqb" podStartSLOduration=0.822582439 podStartE2EDuration="1.275595624s" totalImagesPullingTime="453.013185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:44.274266271 +0000 UTC m=+527.110035489" watchObservedRunningTime="2026-09-17 17:49:44.275595624 +0000 UTC m=+527.111364846"
Sep 17 17:50:59.277683 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:50:59.277657    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:50:59.284556 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:50:59.284536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:53:32.811580 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.811546    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"]
Sep 17 17:53:32.814556 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.814541    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:32.817447 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.817422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-gk487\""
Sep 17 17:53:32.817567 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.817425    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 17:53:32.817567 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.817480    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 17:53:32.817650 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.817564    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-6bkd7\""
Sep 17 17:53:32.817693 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.817685    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 17:53:32.827896 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.827875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"]
Sep 17 17:53:32.928491 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.928462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f75e5ed-1980-4347-856d-65f580029b23-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl\" (UID: \"6f75e5ed-1980-4347-856d-65f580029b23\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:32.928628 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:32.928501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqp4\" (UniqueName: \"kubernetes.io/projected/6f75e5ed-1980-4347-856d-65f580029b23-kube-api-access-fvqp4\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl\" (UID: \"6f75e5ed-1980-4347-856d-65f580029b23\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:33.029716 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:33.029688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f75e5ed-1980-4347-856d-65f580029b23-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl\" (UID: \"6f75e5ed-1980-4347-856d-65f580029b23\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:33.029857 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:33.029720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqp4\" (UniqueName: \"kubernetes.io/projected/6f75e5ed-1980-4347-856d-65f580029b23-kube-api-access-fvqp4\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl\" (UID: \"6f75e5ed-1980-4347-856d-65f580029b23\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:33.032210 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:33.032181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f75e5ed-1980-4347-856d-65f580029b23-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl\" (UID: \"6f75e5ed-1980-4347-856d-65f580029b23\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:33.038749 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:33.038730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqp4\" (UniqueName: \"kubernetes.io/projected/6f75e5ed-1980-4347-856d-65f580029b23-kube-api-access-fvqp4\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl\" (UID: \"6f75e5ed-1980-4347-856d-65f580029b23\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:33.124327 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:33.124242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:33.398349 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:33.398168    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"]
Sep 17 17:53:33.401029 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:53:33.400992    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f75e5ed_1980_4347_856d_65f580029b23.slice/crio-b86119747eb0b0c9b2b308d9850e0961d9d3ab9c60375366402bbfe17bde8733 WatchSource:0}: Error finding container b86119747eb0b0c9b2b308d9850e0961d9d3ab9c60375366402bbfe17bde8733: Status 404 returned error can't find the container with id b86119747eb0b0c9b2b308d9850e0961d9d3ab9c60375366402bbfe17bde8733
Sep 17 17:53:33.403083 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:33.403061    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:53:34.073218 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:34.073141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" event={"ID":"6f75e5ed-1980-4347-856d-65f580029b23","Type":"ContainerStarted","Data":"b86119747eb0b0c9b2b308d9850e0961d9d3ab9c60375366402bbfe17bde8733"}
Sep 17 17:53:37.086864 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:37.086826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" event={"ID":"6f75e5ed-1980-4347-856d-65f580029b23","Type":"ContainerStarted","Data":"f90f59bff7b5e4aaa6be00d69b31f17c2a35f17ed811ce9d4787a3a08da2bce1"}
Sep 17 17:53:37.087251 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:37.086937    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:53:37.106064 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:37.106002    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" podStartSLOduration=2.21291415 podStartE2EDuration="5.105984475s" totalImagesPullingTime="2.893070325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:37.104866879 +0000 UTC m=+759.940636102" watchObservedRunningTime="2026-09-17 17:53:37.105984475 +0000 UTC m=+759.941753700"
Sep 17 17:53:43.111060 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:43.111026    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc9ca3db-8b65-44b4-a00e-71ff7578fefb" containerID="9aec79c2d3b90c5b6ac79992b681eff043fc179d87bc39525edfb4b247ffc35e" exitCode=0
Sep 17 17:53:43.111592 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:43.111107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" event={"ID":"fc9ca3db-8b65-44b4-a00e-71ff7578fefb","Type":"ContainerDied","Data":"9aec79c2d3b90c5b6ac79992b681eff043fc179d87bc39525edfb4b247ffc35e"}
Sep 17 17:53:43.111665 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:43.111593    2565 scope.go:117] "RemoveContainer" containerID="9aec79c2d3b90c5b6ac79992b681eff043fc179d87bc39525edfb4b247ffc35e"
Sep 17 17:53:43.112555 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:43.112523    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:53:43.112654 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:43.112591    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:53:43.112654 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:43.112628    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:53:44.117051 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:44.117018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" event={"ID":"fc9ca3db-8b65-44b4-a00e-71ff7578fefb","Type":"ContainerStarted","Data":"5faeed7d0d530a06ad5a8564e8aa3ad7571b8b25c77a1d4c668cbfc46de05d98"}
Sep 17 17:53:44.124941 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:44.124924    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:53:44.125104 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:44.125085    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Sep 17 17:53:44.125153 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:44.125136    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" podUID="fc9ca3db-8b65-44b4-a00e-71ff7578fefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Sep 17 17:53:45.124756 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:45.124724    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Sep 17 17:53:45.125117 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:45.124783    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" podUID="fc9ca3db-8b65-44b4-a00e-71ff7578fefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Sep 17 17:53:46.124275 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:46.124241    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Sep 17 17:53:46.124471 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:46.124290    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" podUID="fc9ca3db-8b65-44b4-a00e-71ff7578fefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Sep 17 17:53:47.124612 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:47.124579    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Sep 17 17:53:47.125040 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:47.124663    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" podUID="fc9ca3db-8b65-44b4-a00e-71ff7578fefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Sep 17 17:53:48.124702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:48.124661    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Sep 17 17:53:48.125086 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:48.124718    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc" podUID="fc9ca3db-8b65-44b4-a00e-71ff7578fefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Sep 17 17:53:49.129451 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:49.129420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:53:49.129837 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:49.129758    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:53:49.130522 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:49.130490    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc"
Sep 17 17:53:58.640039 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.640005    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"]
Sep 17 17:53:58.647020 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.646999    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:53:58.650055 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.650024    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 17:53:58.650887 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.650841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-h9zzl\""
Sep 17 17:53:58.651011 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.650989    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-pjtsc\""
Sep 17 17:53:58.651183 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.651164    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 17:53:58.651293 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.651244    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 17:53:58.652159 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.652139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"]
Sep 17 17:53:58.773148 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.773117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12490ee3-374d-4c15-a314-7e018e2c8fd1-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj\" (UID: \"12490ee3-374d-4c15-a314-7e018e2c8fd1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:53:58.773326 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.773187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdm86\" (UniqueName: \"kubernetes.io/projected/12490ee3-374d-4c15-a314-7e018e2c8fd1-kube-api-access-qdm86\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj\" (UID: \"12490ee3-374d-4c15-a314-7e018e2c8fd1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:53:58.873717 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.873682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdm86\" (UniqueName: \"kubernetes.io/projected/12490ee3-374d-4c15-a314-7e018e2c8fd1-kube-api-access-qdm86\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj\" (UID: \"12490ee3-374d-4c15-a314-7e018e2c8fd1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:53:58.873932 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.873723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12490ee3-374d-4c15-a314-7e018e2c8fd1-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj\" (UID: \"12490ee3-374d-4c15-a314-7e018e2c8fd1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:53:58.876299 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.876271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12490ee3-374d-4c15-a314-7e018e2c8fd1-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj\" (UID: \"12490ee3-374d-4c15-a314-7e018e2c8fd1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:53:58.880875 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.880851    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdm86\" (UniqueName: \"kubernetes.io/projected/12490ee3-374d-4c15-a314-7e018e2c8fd1-kube-api-access-qdm86\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj\" (UID: \"12490ee3-374d-4c15-a314-7e018e2c8fd1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:53:58.957676 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:58.957652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:53:59.226545 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:53:59.226451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"]
Sep 17 17:53:59.230735 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:53:59.230708    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12490ee3_374d_4c15_a314_7e018e2c8fd1.slice/crio-aae9d2c1be4f16dc595a021cb025604fb2c01a89728295faf5f6f208e371e69a WatchSource:0}: Error finding container aae9d2c1be4f16dc595a021cb025604fb2c01a89728295faf5f6f208e371e69a: Status 404 returned error can't find the container with id aae9d2c1be4f16dc595a021cb025604fb2c01a89728295faf5f6f208e371e69a
Sep 17 17:54:00.178744 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:00.178712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" event={"ID":"12490ee3-374d-4c15-a314-7e018e2c8fd1","Type":"ContainerStarted","Data":"33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371"}
Sep 17 17:54:00.179155 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:00.178762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" event={"ID":"12490ee3-374d-4c15-a314-7e018e2c8fd1","Type":"ContainerStarted","Data":"aae9d2c1be4f16dc595a021cb025604fb2c01a89728295faf5f6f208e371e69a"}
Sep 17 17:54:00.179155 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:00.178808    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:54:00.195996 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:00.195952    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" podStartSLOduration=2.195937503 podStartE2EDuration="2.195937503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:00.194616492 +0000 UTC m=+783.030385718" watchObservedRunningTime="2026-09-17 17:54:00.195937503 +0000 UTC m=+783.031706725"
Sep 17 17:54:08.092216 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:08.092178    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:17.237923 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:17.237834    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"]
Sep 17 17:54:17.239563 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:17.239527    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:27.239546 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:27.239509    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:31.185239 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:31.185209    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:54:32.257153 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:32.257111    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main" containerID="cri-o://f90f59bff7b5e4aaa6be00d69b31f17c2a35f17ed811ce9d4787a3a08da2bce1" gracePeriod=45
Sep 17 17:54:37.239736 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:37.239700    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:38.229260 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:38.229225    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"]
Sep 17 17:54:41.185527 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:41.185492    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:47.240050 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:47.240016    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:51.185645 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:51.185607    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:53.248430 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:53.248357    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main" containerID="cri-o://33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371" gracePeriod=45
Sep 17 17:54:57.240342 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:54:57.240303    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:55:01.185369 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:01.185326    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:55:01.185861 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:01.185462    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:55:02.409609 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.409571    2565 generic.go:356] "Generic (PLEG): container finished" podID="6f75e5ed-1980-4347-856d-65f580029b23" containerID="f90f59bff7b5e4aaa6be00d69b31f17c2a35f17ed811ce9d4787a3a08da2bce1" exitCode=0
Sep 17 17:55:02.409978 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.409586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" event={"ID":"6f75e5ed-1980-4347-856d-65f580029b23","Type":"ContainerDied","Data":"f90f59bff7b5e4aaa6be00d69b31f17c2a35f17ed811ce9d4787a3a08da2bce1"}
Sep 17 17:55:02.500119 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.500096    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:55:02.583940 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.583857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f75e5ed-1980-4347-856d-65f580029b23-tls-certs\") pod \"6f75e5ed-1980-4347-856d-65f580029b23\" (UID: \"6f75e5ed-1980-4347-856d-65f580029b23\") "
Sep 17 17:55:02.583940 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.583898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvqp4\" (UniqueName: \"kubernetes.io/projected/6f75e5ed-1980-4347-856d-65f580029b23-kube-api-access-fvqp4\") pod \"6f75e5ed-1980-4347-856d-65f580029b23\" (UID: \"6f75e5ed-1980-4347-856d-65f580029b23\") "
Sep 17 17:55:02.586085 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.586060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f75e5ed-1980-4347-856d-65f580029b23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f75e5ed-1980-4347-856d-65f580029b23" (UID: "6f75e5ed-1980-4347-856d-65f580029b23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:02.586194 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.586129    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f75e5ed-1980-4347-856d-65f580029b23-kube-api-access-fvqp4" (OuterVolumeSpecName: "kube-api-access-fvqp4") pod "6f75e5ed-1980-4347-856d-65f580029b23" (UID: "6f75e5ed-1980-4347-856d-65f580029b23"). InnerVolumeSpecName "kube-api-access-fvqp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:02.684805 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.684768    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f75e5ed-1980-4347-856d-65f580029b23-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:55:02.684805 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:02.684797    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqp4\" (UniqueName: \"kubernetes.io/projected/6f75e5ed-1980-4347-856d-65f580029b23-kube-api-access-fvqp4\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:55:03.414112 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:03.414080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl" event={"ID":"6f75e5ed-1980-4347-856d-65f580029b23","Type":"ContainerDied","Data":"b86119747eb0b0c9b2b308d9850e0961d9d3ab9c60375366402bbfe17bde8733"}
Sep 17 17:55:03.414498 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:03.414115    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"
Sep 17 17:55:03.414498 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:03.414134    2565 scope.go:117] "RemoveContainer" containerID="f90f59bff7b5e4aaa6be00d69b31f17c2a35f17ed811ce9d4787a3a08da2bce1"
Sep 17 17:55:03.430971 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:03.430946    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"]
Sep 17 17:55:03.434072 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:03.434052    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5h7lsl"]
Sep 17 17:55:05.348683 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:05.348640    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f75e5ed-1980-4347-856d-65f580029b23" path="/var/lib/kubelet/pods/6f75e5ed-1980-4347-856d-65f580029b23/volumes"
Sep 17 17:55:09.505124 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.505098    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"]
Sep 17 17:55:09.505553 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.505504    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main"
Sep 17 17:55:09.505553 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.505521    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main"
Sep 17 17:55:09.505641 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.505592    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f75e5ed-1980-4347-856d-65f580029b23" containerName="main"
Sep 17 17:55:09.514988 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.514964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.517236 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.517217    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 17:55:09.518757 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.518740    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 17:55:09.519367 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.519352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-mn96v\""
Sep 17 17:55:09.519475 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.519363    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 17:55:09.519475 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.519363    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-c92fr\""
Sep 17 17:55:09.524559 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.524536    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"]
Sep 17 17:55:09.636934 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.636898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1f46d-a744-41de-9833-319cf20c2432-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.637099 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.636943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txd96\" (UniqueName: \"kubernetes.io/projected/12b1f46d-a744-41de-9833-319cf20c2432-kube-api-access-txd96\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.637099 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.636964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.637099 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.637048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.737967 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.737925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.738152 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.738017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1f46d-a744-41de-9833-319cf20c2432-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.738152 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.738062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txd96\" (UniqueName: \"kubernetes.io/projected/12b1f46d-a744-41de-9833-319cf20c2432-kube-api-access-txd96\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.738152 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.738087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.738331 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.738307    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.738372 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.738346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.740565 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.740540    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1f46d-a744-41de-9833-319cf20c2432-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.747442 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.747417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txd96\" (UniqueName: \"kubernetes.io/projected/12b1f46d-a744-41de-9833-319cf20c2432-kube-api-access-txd96\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:09.825749 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:09.825677    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:10.102698 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:10.102618    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"]
Sep 17 17:55:10.105711 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:55:10.105678    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b1f46d_a744_41de_9833_319cf20c2432.slice/crio-ce55fe16d195e503e7cca984e107a098e460ba6de3aebd9fcd2616c5d8d35ce9 WatchSource:0}: Error finding container ce55fe16d195e503e7cca984e107a098e460ba6de3aebd9fcd2616c5d8d35ce9: Status 404 returned error can't find the container with id ce55fe16d195e503e7cca984e107a098e460ba6de3aebd9fcd2616c5d8d35ce9
Sep 17 17:55:10.442237 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:10.442199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" event={"ID":"12b1f46d-a744-41de-9833-319cf20c2432","Type":"ContainerStarted","Data":"5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf"}
Sep 17 17:55:10.442438 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:10.442262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" event={"ID":"12b1f46d-a744-41de-9833-319cf20c2432","Type":"ContainerStarted","Data":"ce55fe16d195e503e7cca984e107a098e460ba6de3aebd9fcd2616c5d8d35ce9"}
Sep 17 17:55:10.442438 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:10.442290    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:10.472792 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:10.472732    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" podStartSLOduration=1.472712437 podStartE2EDuration="1.472712437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:10.470998184 +0000 UTC m=+853.306767408" watchObservedRunningTime="2026-09-17 17:55:10.472712437 +0000 UTC m=+853.308481664"
Sep 17 17:55:11.186074 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:11.186037    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:55:21.185600 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:21.185564    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:55:23.489910 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.489886    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:55:23.493154 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.493128    2565 generic.go:356] "Generic (PLEG): container finished" podID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerID="33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371" exitCode=0
Sep 17 17:55:23.493266 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.493159    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"
Sep 17 17:55:23.493266 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.493184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" event={"ID":"12490ee3-374d-4c15-a314-7e018e2c8fd1","Type":"ContainerDied","Data":"33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371"}
Sep 17 17:55:23.493266 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.493223    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj" event={"ID":"12490ee3-374d-4c15-a314-7e018e2c8fd1","Type":"ContainerDied","Data":"aae9d2c1be4f16dc595a021cb025604fb2c01a89728295faf5f6f208e371e69a"}
Sep 17 17:55:23.493266 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.493243    2565 scope.go:117] "RemoveContainer" containerID="33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371"
Sep 17 17:55:23.500660 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.500640    2565 scope.go:117] "RemoveContainer" containerID="33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371"
Sep 17 17:55:23.500938 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:55:23.500920    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371\": container with ID starting with 33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371 not found: ID does not exist" containerID="33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371"
Sep 17 17:55:23.500983 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.500946    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371"} err="failed to get container status \"33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371\": rpc error: code = NotFound desc = could not find container \"33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371\": container with ID starting with 33ce34424cedec38245ff8a72cefa437e51cd7b9831cdc3c0b1db68378d53371 not found: ID does not exist"
Sep 17 17:55:23.644366 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.644297    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdm86\" (UniqueName: \"kubernetes.io/projected/12490ee3-374d-4c15-a314-7e018e2c8fd1-kube-api-access-qdm86\") pod \"12490ee3-374d-4c15-a314-7e018e2c8fd1\" (UID: \"12490ee3-374d-4c15-a314-7e018e2c8fd1\") "
Sep 17 17:55:23.644366 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.644333    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12490ee3-374d-4c15-a314-7e018e2c8fd1-tls-certs\") pod \"12490ee3-374d-4c15-a314-7e018e2c8fd1\" (UID: \"12490ee3-374d-4c15-a314-7e018e2c8fd1\") "
Sep 17 17:55:23.646494 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.646462    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12490ee3-374d-4c15-a314-7e018e2c8fd1-kube-api-access-qdm86" (OuterVolumeSpecName: "kube-api-access-qdm86") pod "12490ee3-374d-4c15-a314-7e018e2c8fd1" (UID: "12490ee3-374d-4c15-a314-7e018e2c8fd1"). InnerVolumeSpecName "kube-api-access-qdm86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:23.646494 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.646491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12490ee3-374d-4c15-a314-7e018e2c8fd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12490ee3-374d-4c15-a314-7e018e2c8fd1" (UID: "12490ee3-374d-4c15-a314-7e018e2c8fd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:23.745167 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.745139    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdm86\" (UniqueName: \"kubernetes.io/projected/12490ee3-374d-4c15-a314-7e018e2c8fd1-kube-api-access-qdm86\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:55:23.745167 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.745163    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12490ee3-374d-4c15-a314-7e018e2c8fd1-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:55:23.815848 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.815820    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"]
Sep 17 17:55:23.820189 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:23.820170    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-sxngj"]
Sep 17 17:55:25.348677 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:25.348647    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" path="/var/lib/kubelet/pods/12490ee3-374d-4c15-a314-7e018e2c8fd1/volumes"
Sep 17 17:55:41.448388 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:41.448314    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:55:59.303912 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:59.303888    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:55:59.306064 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:55:59.306040    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 17:56:04.109830 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.109798    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"]
Sep 17 17:56:04.110185 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.110176    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main"
Sep 17 17:56:04.110224 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.110191    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main"
Sep 17 17:56:04.110288 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.110276    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="12490ee3-374d-4c15-a314-7e018e2c8fd1" containerName="main"
Sep 17 17:56:04.114281 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.114263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:04.117144 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.117105    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 17:56:04.117259 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.117141    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 17:56:04.117259 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.117108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-xkxgr\""
Sep 17 17:56:04.117259 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.117205    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-5vnpl\""
Sep 17 17:56:04.117463 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.117310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 17:56:04.122532 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.122502    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"]
Sep 17 17:56:04.154245 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.154218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkldw\" (UniqueName: \"kubernetes.io/projected/4e4a1992-8623-450d-ba0c-cac69526d175-kube-api-access-xkldw\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f\" (UID: \"4e4a1992-8623-450d-ba0c-cac69526d175\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:04.154376 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.154280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4a1992-8623-450d-ba0c-cac69526d175-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f\" (UID: \"4e4a1992-8623-450d-ba0c-cac69526d175\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:04.254907 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.254875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4a1992-8623-450d-ba0c-cac69526d175-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f\" (UID: \"4e4a1992-8623-450d-ba0c-cac69526d175\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:04.255081 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.254925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkldw\" (UniqueName: \"kubernetes.io/projected/4e4a1992-8623-450d-ba0c-cac69526d175-kube-api-access-xkldw\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f\" (UID: \"4e4a1992-8623-450d-ba0c-cac69526d175\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:04.257719 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.257691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4a1992-8623-450d-ba0c-cac69526d175-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f\" (UID: \"4e4a1992-8623-450d-ba0c-cac69526d175\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:04.263684 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.263658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkldw\" (UniqueName: \"kubernetes.io/projected/4e4a1992-8623-450d-ba0c-cac69526d175-kube-api-access-xkldw\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f\" (UID: \"4e4a1992-8623-450d-ba0c-cac69526d175\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:04.425735 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.425707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:04.698864 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:04.698784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"]
Sep 17 17:56:04.702046 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:56:04.702012    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4a1992_8623_450d_ba0c_cac69526d175.slice/crio-d8ef17b16dabbcaad5413a37da9d2729ad32ed17513566744cb2e40d2d25851a WatchSource:0}: Error finding container d8ef17b16dabbcaad5413a37da9d2729ad32ed17513566744cb2e40d2d25851a: Status 404 returned error can't find the container with id d8ef17b16dabbcaad5413a37da9d2729ad32ed17513566744cb2e40d2d25851a
Sep 17 17:56:05.668134 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:05.668099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" event={"ID":"4e4a1992-8623-450d-ba0c-cac69526d175","Type":"ContainerStarted","Data":"beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d"}
Sep 17 17:56:05.668542 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:05.668155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" event={"ID":"4e4a1992-8623-450d-ba0c-cac69526d175","Type":"ContainerStarted","Data":"d8ef17b16dabbcaad5413a37da9d2729ad32ed17513566744cb2e40d2d25851a"}
Sep 17 17:56:05.668542 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:05.668218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:05.686221 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:05.686174    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" podStartSLOduration=1.686159951 podStartE2EDuration="1.686159951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:05.684995861 +0000 UTC m=+908.520765087" watchObservedRunningTime="2026-09-17 17:56:05.686159951 +0000 UTC m=+908.521929175"
Sep 17 17:56:06.741682 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:06.741650    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"]
Sep 17 17:56:11.449060 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:11.449018    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:21.448467 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:21.448428    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:21.760554 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:21.760453    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main" containerID="cri-o://5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf" gracePeriod=45
Sep 17 17:56:31.447907 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:31.447870    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:31.448290 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:31.447965    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:56:36.673765 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:36.673741    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:56:41.448688 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:41.448649    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:43.574809 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:43.574779    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"]
Sep 17 17:56:46.673835 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:46.673798    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:51.448534 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:51.448493    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:52.005780 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.005758    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:56:52.038998 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.038934    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txd96\" (UniqueName: \"kubernetes.io/projected/12b1f46d-a744-41de-9833-319cf20c2432-kube-api-access-txd96\") pod \"12b1f46d-a744-41de-9833-319cf20c2432\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") "
Sep 17 17:56:52.038998 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.038966    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-cachi2-cache\") pod \"12b1f46d-a744-41de-9833-319cf20c2432\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") "
Sep 17 17:56:52.038998 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.038999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1f46d-a744-41de-9833-319cf20c2432-tls-certs\") pod \"12b1f46d-a744-41de-9833-319cf20c2432\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") "
Sep 17 17:56:52.039187 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.039062    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-tokenizer-cache\") pod \"12b1f46d-a744-41de-9833-319cf20c2432\" (UID: \"12b1f46d-a744-41de-9833-319cf20c2432\") "
Sep 17 17:56:52.039266 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.039237    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "12b1f46d-a744-41de-9833-319cf20c2432" (UID: "12b1f46d-a744-41de-9833-319cf20c2432"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:52.039418 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.039377    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "12b1f46d-a744-41de-9833-319cf20c2432" (UID: "12b1f46d-a744-41de-9833-319cf20c2432"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:52.041233 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.041214    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b1f46d-a744-41de-9833-319cf20c2432-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12b1f46d-a744-41de-9833-319cf20c2432" (UID: "12b1f46d-a744-41de-9833-319cf20c2432"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:52.041302 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.041246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b1f46d-a744-41de-9833-319cf20c2432-kube-api-access-txd96" (OuterVolumeSpecName: "kube-api-access-txd96") pod "12b1f46d-a744-41de-9833-319cf20c2432" (UID: "12b1f46d-a744-41de-9833-319cf20c2432"). InnerVolumeSpecName "kube-api-access-txd96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:52.139979 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.139937    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-tokenizer-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:56:52.139979 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.139973    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txd96\" (UniqueName: \"kubernetes.io/projected/12b1f46d-a744-41de-9833-319cf20c2432-kube-api-access-txd96\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:56:52.139979 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.139987    2565 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b1f46d-a744-41de-9833-319cf20c2432-cachi2-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:56:52.140211 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.139999    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1f46d-a744-41de-9833-319cf20c2432-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:56:52.840923 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.840882    2565 generic.go:356] "Generic (PLEG): container finished" podID="12b1f46d-a744-41de-9833-319cf20c2432" containerID="5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf" exitCode=0
Sep 17 17:56:52.841413 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.840934    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"
Sep 17 17:56:52.841413 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.840970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" event={"ID":"12b1f46d-a744-41de-9833-319cf20c2432","Type":"ContainerDied","Data":"5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf"}
Sep 17 17:56:52.841413 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.841020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd" event={"ID":"12b1f46d-a744-41de-9833-319cf20c2432","Type":"ContainerDied","Data":"ce55fe16d195e503e7cca984e107a098e460ba6de3aebd9fcd2616c5d8d35ce9"}
Sep 17 17:56:52.841413 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.841041    2565 scope.go:117] "RemoveContainer" containerID="5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf"
Sep 17 17:56:52.850179 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.850161    2565 scope.go:117] "RemoveContainer" containerID="5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf"
Sep 17 17:56:52.850512 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:56:52.850493    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf\": container with ID starting with 5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf not found: ID does not exist" containerID="5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf"
Sep 17 17:56:52.850582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.850520    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf"} err="failed to get container status \"5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf\": rpc error: code = NotFound desc = could not find container \"5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf\": container with ID starting with 5da081c284e8ada500aae77a997a06c8c89781ed10c25902b616c3d770dffedf not found: ID does not exist"
Sep 17 17:56:52.865484 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.865462    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"]
Sep 17 17:56:52.870893 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:52.870872    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcvwxzd"]
Sep 17 17:56:53.347819 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:53.347785    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b1f46d-a744-41de-9833-319cf20c2432" path="/var/lib/kubelet/pods/12b1f46d-a744-41de-9833-319cf20c2432/volumes"
Sep 17 17:56:56.674346 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:56.674310    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:58.593604 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:56:58.593538    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main" containerID="cri-o://beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d" gracePeriod=45
Sep 17 17:57:01.297874 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.297844    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"]
Sep 17 17:57:01.298224 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.298105    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main"
Sep 17 17:57:01.298224 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.298114    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main"
Sep 17 17:57:01.298224 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.298161    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b1f46d-a744-41de-9833-319cf20c2432" containerName="main"
Sep 17 17:57:01.302168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.302148    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:01.305076 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.305054    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 17:57:01.305187 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.305052    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.305187 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.305057    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.305187 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.305144    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-bc27z\""
Sep 17 17:57:01.305187 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.305104    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-nsgbc\""
Sep 17 17:57:01.312950 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.312930    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"]
Sep 17 17:57:01.412802 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.412775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4z4\" (UniqueName: \"kubernetes.io/projected/001270b0-71a2-4ef8-8c16-fe0bef7702a3-kube-api-access-rz4z4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl\" (UID: \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:01.412978 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.412808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001270b0-71a2-4ef8-8c16-fe0bef7702a3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl\" (UID: \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:01.513888 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.513854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4z4\" (UniqueName: \"kubernetes.io/projected/001270b0-71a2-4ef8-8c16-fe0bef7702a3-kube-api-access-rz4z4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl\" (UID: \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:01.513888 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.513891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001270b0-71a2-4ef8-8c16-fe0bef7702a3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl\" (UID: \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:01.516326 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.516300    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001270b0-71a2-4ef8-8c16-fe0bef7702a3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl\" (UID: \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:01.522107 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.522086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4z4\" (UniqueName: \"kubernetes.io/projected/001270b0-71a2-4ef8-8c16-fe0bef7702a3-kube-api-access-rz4z4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl\" (UID: \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:01.612039 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.611981    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:01.885608 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:01.885586    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"]
Sep 17 17:57:01.888315 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:57:01.888286    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001270b0_71a2_4ef8_8c16_fe0bef7702a3.slice/crio-8b82d36760cf6579608330495fb4365cec4e1d28d2f73f4a0b2fa75f81ffac51 WatchSource:0}: Error finding container 8b82d36760cf6579608330495fb4365cec4e1d28d2f73f4a0b2fa75f81ffac51: Status 404 returned error can't find the container with id 8b82d36760cf6579608330495fb4365cec4e1d28d2f73f4a0b2fa75f81ffac51
Sep 17 17:57:02.881662 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:02.881623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" event={"ID":"001270b0-71a2-4ef8-8c16-fe0bef7702a3","Type":"ContainerStarted","Data":"992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639"}
Sep 17 17:57:02.882071 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:02.881704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" event={"ID":"001270b0-71a2-4ef8-8c16-fe0bef7702a3","Type":"ContainerStarted","Data":"8b82d36760cf6579608330495fb4365cec4e1d28d2f73f4a0b2fa75f81ffac51"}
Sep 17 17:57:02.882071 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:02.881730    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:02.900968 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:02.900922    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" podStartSLOduration=1.900907752 podStartE2EDuration="1.900907752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:02.899906776 +0000 UTC m=+965.735676000" watchObservedRunningTime="2026-09-17 17:57:02.900907752 +0000 UTC m=+965.736676975"
Sep 17 17:57:06.674458 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:06.674424    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:57:06.674829 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:06.674529    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:57:16.674496 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:16.674384    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:57:26.673919 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:26.673877    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:57:28.832231 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.832205    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:57:28.928059 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.928023    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkldw\" (UniqueName: \"kubernetes.io/projected/4e4a1992-8623-450d-ba0c-cac69526d175-kube-api-access-xkldw\") pod \"4e4a1992-8623-450d-ba0c-cac69526d175\" (UID: \"4e4a1992-8623-450d-ba0c-cac69526d175\") "
Sep 17 17:57:28.928228 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.928096    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4a1992-8623-450d-ba0c-cac69526d175-tls-certs\") pod \"4e4a1992-8623-450d-ba0c-cac69526d175\" (UID: \"4e4a1992-8623-450d-ba0c-cac69526d175\") "
Sep 17 17:57:28.930296 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.930261    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4a1992-8623-450d-ba0c-cac69526d175-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e4a1992-8623-450d-ba0c-cac69526d175" (UID: "4e4a1992-8623-450d-ba0c-cac69526d175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:28.930389 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.930268    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4a1992-8623-450d-ba0c-cac69526d175-kube-api-access-xkldw" (OuterVolumeSpecName: "kube-api-access-xkldw") pod "4e4a1992-8623-450d-ba0c-cac69526d175" (UID: "4e4a1992-8623-450d-ba0c-cac69526d175"). InnerVolumeSpecName "kube-api-access-xkldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:28.980261 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.980226    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e4a1992-8623-450d-ba0c-cac69526d175" containerID="beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d" exitCode=0
Sep 17 17:57:28.980471 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.980268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"
Sep 17 17:57:28.980471 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.980308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" event={"ID":"4e4a1992-8623-450d-ba0c-cac69526d175","Type":"ContainerDied","Data":"beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d"}
Sep 17 17:57:28.980471 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.980356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f" event={"ID":"4e4a1992-8623-450d-ba0c-cac69526d175","Type":"ContainerDied","Data":"d8ef17b16dabbcaad5413a37da9d2729ad32ed17513566744cb2e40d2d25851a"}
Sep 17 17:57:28.980471 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.980379    2565 scope.go:117] "RemoveContainer" containerID="beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d"
Sep 17 17:57:28.988722 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.988707    2565 scope.go:117] "RemoveContainer" containerID="beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d"
Sep 17 17:57:28.988982 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:57:28.988964    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d\": container with ID starting with beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d not found: ID does not exist" containerID="beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d"
Sep 17 17:57:28.989026 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:28.988990    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d"} err="failed to get container status \"beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d\": rpc error: code = NotFound desc = could not find container \"beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d\": container with ID starting with beaeb711225119a74e6d5b750b12fad57aefe3cd1432c4dc5e3caf047fb3d56d not found: ID does not exist"
Sep 17 17:57:29.001000 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:29.000976    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"]
Sep 17 17:57:29.005226 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:29.005207    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fqb6f"]
Sep 17 17:57:29.029269 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:29.029247    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkldw\" (UniqueName: \"kubernetes.io/projected/4e4a1992-8623-450d-ba0c-cac69526d175-kube-api-access-xkldw\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:29.029269 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:29.029268    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4a1992-8623-450d-ba0c-cac69526d175-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:29.347899 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:29.347823    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" path="/var/lib/kubelet/pods/4e4a1992-8623-450d-ba0c-cac69526d175/volumes"
Sep 17 17:57:33.702039 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.702007    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"]
Sep 17 17:57:33.702543 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.702383    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main"
Sep 17 17:57:33.702543 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.702419    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main"
Sep 17 17:57:33.702543 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.702517    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4a1992-8623-450d-ba0c-cac69526d175" containerName="main"
Sep 17 17:57:33.706575 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.706555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.710163 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.710138    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:57:33.710352 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.710336    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 17:57:33.710535 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.710508    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-jlwcl\""
Sep 17 17:57:33.713934 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.713915    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 17:57:33.723849 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.723827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"]
Sep 17 17:57:33.864358 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.864358 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.864593 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.864593 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlclb\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-kube-api-access-xlclb\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.864593 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093dba64-0665-40d2-a081-2be82d7aafdf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.864593 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.864751 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.864751 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093dba64-0665-40d2-a081-2be82d7aafdf-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.864751 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.864690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.887438 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.887416    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:57:33.965795 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.965709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.965795 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.965744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.965795 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.965775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093dba64-0665-40d2-a081-2be82d7aafdf-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.965795 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.965796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966105 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.965825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966105 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.965838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966105 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.965861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966105 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.965884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlclb\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-kube-api-access-xlclb\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966295 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.966204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966295 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.966227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966417 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.966319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966417 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.966339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093dba64-0665-40d2-a081-2be82d7aafdf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.966561 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.966535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.967071 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.967046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093dba64-0665-40d2-a081-2be82d7aafdf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.968102 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.968085    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.968686 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.968661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093dba64-0665-40d2-a081-2be82d7aafdf-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.975315 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.975282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlclb\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-kube-api-access-xlclb\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:33.975315 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:33.975308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-cmqm6\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:34.019628 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:34.019599    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:34.299208 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:34.299127    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"]
Sep 17 17:57:34.302097 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:57:34.302065    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093dba64_0665_40d2_a081_2be82d7aafdf.slice/crio-f2b38673fc7f3d9f8ef038cb8df82a5504a861941549e66ca46c48decd569af5 WatchSource:0}: Error finding container f2b38673fc7f3d9f8ef038cb8df82a5504a861941549e66ca46c48decd569af5: Status 404 returned error can't find the container with id f2b38673fc7f3d9f8ef038cb8df82a5504a861941549e66ca46c48decd569af5
Sep 17 17:57:34.304453 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:34.304419    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:57:34.304545 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:34.304485    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:57:34.304545 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:34.304517    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:57:35.004576 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:35.004543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6" event={"ID":"093dba64-0665-40d2-a081-2be82d7aafdf","Type":"ContainerStarted","Data":"e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039"}
Sep 17 17:57:35.004930 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:35.004601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6" event={"ID":"093dba64-0665-40d2-a081-2be82d7aafdf","Type":"ContainerStarted","Data":"f2b38673fc7f3d9f8ef038cb8df82a5504a861941549e66ca46c48decd569af5"}
Sep 17 17:57:35.020389 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:35.020368    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:35.025613 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:35.025593    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:35.030743 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:35.030702    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6" podStartSLOduration=2.030686607 podStartE2EDuration="2.030686607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:35.02952131 +0000 UTC m=+997.865290532" watchObservedRunningTime="2026-09-17 17:57:35.030686607 +0000 UTC m=+997.866455830"
Sep 17 17:57:36.008718 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:36.008685    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:36.009607 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:36.009585    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:42.580126 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.580087    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"]
Sep 17 17:57:42.583242 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.583226    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.585814 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.585794    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 17:57:42.585905 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.585824    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-ghb46\""
Sep 17 17:57:42.594800 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.594779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"]
Sep 17 17:57:42.741437 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.741388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.741582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.741444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ade6400-1078-4f31-a0d2-c136ad16c90b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.741582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.741477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.741582 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.741532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.741684 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.741593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.741684 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.741631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tdc\" (UniqueName: \"kubernetes.io/projected/6ade6400-1078-4f31-a0d2-c136ad16c90b-kube-api-access-c5tdc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842358 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842358 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tdc\" (UniqueName: \"kubernetes.io/projected/6ade6400-1078-4f31-a0d2-c136ad16c90b-kube-api-access-c5tdc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842358 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842642 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ade6400-1078-4f31-a0d2-c136ad16c90b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842642 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842642 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842795 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842795 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.842898 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.842868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.844625 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.844601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.844878 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.844861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ade6400-1078-4f31-a0d2-c136ad16c90b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.865565 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.865536    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tdc\" (UniqueName: \"kubernetes.io/projected/6ade6400-1078-4f31-a0d2-c136ad16c90b-kube-api-access-c5tdc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:42.893402 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:42.893367    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:43.169047 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:43.169013    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"]
Sep 17 17:57:43.172449 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:57:43.172422    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ade6400_1078_4f31_a0d2_c136ad16c90b.slice/crio-a4f4a1ad658626e76362d7b95c9ff067213a5bfe2850b59c32662185c92fac88 WatchSource:0}: Error finding container a4f4a1ad658626e76362d7b95c9ff067213a5bfe2850b59c32662185c92fac88: Status 404 returned error can't find the container with id a4f4a1ad658626e76362d7b95c9ff067213a5bfe2850b59c32662185c92fac88
Sep 17 17:57:44.038187 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:44.038147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7" event={"ID":"6ade6400-1078-4f31-a0d2-c136ad16c90b","Type":"ContainerStarted","Data":"a4f4a1ad658626e76362d7b95c9ff067213a5bfe2850b59c32662185c92fac88"}
Sep 17 17:57:46.049777 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:46.049728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7" event={"ID":"6ade6400-1078-4f31-a0d2-c136ad16c90b","Type":"ContainerStarted","Data":"ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482"}
Sep 17 17:57:46.082498 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:46.082423    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7" podStartSLOduration=2.186967299 podStartE2EDuration="4.082376674s" totalImagesPullingTime="1.895409375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:46.081947153 +0000 UTC m=+1008.917716374" watchObservedRunningTime="2026-09-17 17:57:46.082376674 +0000 UTC m=+1008.918145899"
Sep 17 17:57:46.894116 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:46.894075    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:49.089756 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.089725    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"]
Sep 17 17:57:49.098163 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:57:49.098127    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:57:49.098163 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:49.098163 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:49.098163 ip-10-0-129-186 kubenswrapper[2565]:  > containerID="ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482" cmd=["/bin/sleep","15"]
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.098177    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7" message="" err=<
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]:  >
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:57:49.098190    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7" podUID="6ade6400-1078-4f31-a0d2-c136ad16c90b" containerName="main" containerID="cri-o://ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482"
Sep 17 17:57:49.098375 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.098231    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7" podUID="6ade6400-1078-4f31-a0d2-c136ad16c90b" containerName="main" containerID="cri-o://ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482" gracePeriod=60
Sep 17 17:57:49.389672 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.389644    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:49.408881 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.408857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-dshm\") pod \"6ade6400-1078-4f31-a0d2-c136ad16c90b\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") "
Sep 17 17:57:49.409081 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.408927    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-model-cache\") pod \"6ade6400-1078-4f31-a0d2-c136ad16c90b\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") "
Sep 17 17:57:49.409081 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.409008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ade6400-1078-4f31-a0d2-c136ad16c90b-tls-certs\") pod \"6ade6400-1078-4f31-a0d2-c136ad16c90b\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") "
Sep 17 17:57:49.409081 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.409047    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-tmp-dir\") pod \"6ade6400-1078-4f31-a0d2-c136ad16c90b\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") "
Sep 17 17:57:49.409312 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.409081    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5tdc\" (UniqueName: \"kubernetes.io/projected/6ade6400-1078-4f31-a0d2-c136ad16c90b-kube-api-access-c5tdc\") pod \"6ade6400-1078-4f31-a0d2-c136ad16c90b\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") "
Sep 17 17:57:49.409312 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.409202    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-home\") pod \"6ade6400-1078-4f31-a0d2-c136ad16c90b\" (UID: \"6ade6400-1078-4f31-a0d2-c136ad16c90b\") "
Sep 17 17:57:49.409507 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.409415    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ade6400-1078-4f31-a0d2-c136ad16c90b" (UID: "6ade6400-1078-4f31-a0d2-c136ad16c90b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:49.409580 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.409523    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:49.409651 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.409627    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-home" (OuterVolumeSpecName: "home") pod "6ade6400-1078-4f31-a0d2-c136ad16c90b" (UID: "6ade6400-1078-4f31-a0d2-c136ad16c90b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:49.409712 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.409687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ade6400-1078-4f31-a0d2-c136ad16c90b" (UID: "6ade6400-1078-4f31-a0d2-c136ad16c90b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:49.411514 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.411491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-dshm" (OuterVolumeSpecName: "dshm") pod "6ade6400-1078-4f31-a0d2-c136ad16c90b" (UID: "6ade6400-1078-4f31-a0d2-c136ad16c90b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:49.411936 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.411908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ade6400-1078-4f31-a0d2-c136ad16c90b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ade6400-1078-4f31-a0d2-c136ad16c90b" (UID: "6ade6400-1078-4f31-a0d2-c136ad16c90b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:49.411936 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.411919    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ade6400-1078-4f31-a0d2-c136ad16c90b-kube-api-access-c5tdc" (OuterVolumeSpecName: "kube-api-access-c5tdc") pod "6ade6400-1078-4f31-a0d2-c136ad16c90b" (UID: "6ade6400-1078-4f31-a0d2-c136ad16c90b"). InnerVolumeSpecName "kube-api-access-c5tdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:49.510202 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.510163    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5tdc\" (UniqueName: \"kubernetes.io/projected/6ade6400-1078-4f31-a0d2-c136ad16c90b-kube-api-access-c5tdc\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:49.510202 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.510196    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:49.510202 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.510206    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:49.510450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.510214    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ade6400-1078-4f31-a0d2-c136ad16c90b-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:49.510450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:49.510223    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ade6400-1078-4f31-a0d2-c136ad16c90b-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:50.078070 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.078027    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ade6400-1078-4f31-a0d2-c136ad16c90b" containerID="ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482" exitCode=0
Sep 17 17:57:50.078070 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.078073    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"
Sep 17 17:57:50.078329 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.078110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7" event={"ID":"6ade6400-1078-4f31-a0d2-c136ad16c90b","Type":"ContainerDied","Data":"ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482"}
Sep 17 17:57:50.078329 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.078166    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7" event={"ID":"6ade6400-1078-4f31-a0d2-c136ad16c90b","Type":"ContainerDied","Data":"a4f4a1ad658626e76362d7b95c9ff067213a5bfe2850b59c32662185c92fac88"}
Sep 17 17:57:50.078329 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.078191    2565 scope.go:117] "RemoveContainer" containerID="ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482"
Sep 17 17:57:50.088559 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.088538    2565 scope.go:117] "RemoveContainer" containerID="ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482"
Sep 17 17:57:50.088868 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:57:50.088846    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482\": container with ID starting with ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482 not found: ID does not exist" containerID="ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482"
Sep 17 17:57:50.088926 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.088871    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482"} err="failed to get container status \"ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482\": rpc error: code = NotFound desc = could not find container \"ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482\": container with ID starting with ac39840fe6f6a5444209b377a13a3220371031813f7316924718ddc406f3c482 not found: ID does not exist"
Sep 17 17:57:50.101461 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.101437    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"]
Sep 17 17:57:50.106737 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.106710    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"]
Sep 17 17:57:50.106917 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.106896    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6" podUID="093dba64-0665-40d2-a081-2be82d7aafdf" containerName="istio-proxy" containerID="cri-o://e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039" gracePeriod=30
Sep 17 17:57:50.110007 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:50.109984    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-dszn7"]
Sep 17 17:57:51.349378 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:51.349334    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ade6400-1078-4f31-a0d2-c136ad16c90b" path="/var/lib/kubelet/pods/6ade6400-1078-4f31-a0d2-c136ad16c90b/volumes"
Sep 17 17:57:52.050652 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.050589    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6" podUID="093dba64-0665-40d2-a081-2be82d7aafdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 17:57:52.759586 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.759557    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:52.838358 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838330    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093dba64-0665-40d2-a081-2be82d7aafdf-istio-podinfo\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.838521 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838374    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-certs\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.838521 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838424    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-data\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.838521 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838456    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-socket\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.838521 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838488    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093dba64-0665-40d2-a081-2be82d7aafdf-istiod-ca-cert\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.838726 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838529    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlclb\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-kube-api-access-xlclb\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.838726 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838552    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-credential-socket\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.838726 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838585    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-envoy\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.838726 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-istio-token\") pod \"093dba64-0665-40d2-a081-2be82d7aafdf\" (UID: \"093dba64-0665-40d2-a081-2be82d7aafdf\") "
Sep 17 17:57:52.839014 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838748    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:52.839014 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838785    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-data" (OuterVolumeSpecName: "istio-data") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:52.839014 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838924    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:52.839014 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838926    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093dba64-0665-40d2-a081-2be82d7aafdf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:52.839014 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.838943    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:52.839266 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.839013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:52.839266 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.839152    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:52.840803 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.840773    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-kube-api-access-xlclb" (OuterVolumeSpecName: "kube-api-access-xlclb") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "kube-api-access-xlclb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:52.841009 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.840974    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:52.841160 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.841137    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/093dba64-0665-40d2-a081-2be82d7aafdf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 17:57:52.841269 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.841213    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-istio-token" (OuterVolumeSpecName: "istio-token") pod "093dba64-0665-40d2-a081-2be82d7aafdf" (UID: "093dba64-0665-40d2-a081-2be82d7aafdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:52.939441 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.939405    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:52.939441 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.939433    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093dba64-0665-40d2-a081-2be82d7aafdf-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:52.939441 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.939443    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:52.939441 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.939451    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093dba64-0665-40d2-a081-2be82d7aafdf-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:52.939709 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.939461    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlclb\" (UniqueName: \"kubernetes.io/projected/093dba64-0665-40d2-a081-2be82d7aafdf-kube-api-access-xlclb\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:52.939709 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.939469    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:52.939709 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:52.939478    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093dba64-0665-40d2-a081-2be82d7aafdf-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:53.093355 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.093272    2565 generic.go:356] "Generic (PLEG): container finished" podID="093dba64-0665-40d2-a081-2be82d7aafdf" containerID="e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039" exitCode=0
Sep 17 17:57:53.093355 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.093325    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"
Sep 17 17:57:53.093355 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.093331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6" event={"ID":"093dba64-0665-40d2-a081-2be82d7aafdf","Type":"ContainerDied","Data":"e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039"}
Sep 17 17:57:53.093600 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.093373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6" event={"ID":"093dba64-0665-40d2-a081-2be82d7aafdf","Type":"ContainerDied","Data":"f2b38673fc7f3d9f8ef038cb8df82a5504a861941549e66ca46c48decd569af5"}
Sep 17 17:57:53.093600 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.093417    2565 scope.go:117] "RemoveContainer" containerID="e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039"
Sep 17 17:57:53.105604 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.105583    2565 scope.go:117] "RemoveContainer" containerID="e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039"
Sep 17 17:57:53.105907 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:57:53.105887    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039\": container with ID starting with e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039 not found: ID does not exist" containerID="e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039"
Sep 17 17:57:53.105964 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.105919    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039"} err="failed to get container status \"e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039\": rpc error: code = NotFound desc = could not find container \"e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039\": container with ID starting with e836672f5e4253187ca7753c96165dd78f08231cdbc18e8e2fc618a57f2c6039 not found: ID does not exist"
Sep 17 17:57:53.116721 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.116695    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"]
Sep 17 17:57:53.120202 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.120179    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-cmqm6"]
Sep 17 17:57:53.348168 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:53.348097    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093dba64-0665-40d2-a081-2be82d7aafdf" path="/var/lib/kubelet/pods/093dba64-0665-40d2-a081-2be82d7aafdf/volumes"
Sep 17 17:57:54.296287 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.296250    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"]
Sep 17 17:57:54.296754 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.296670    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093dba64-0665-40d2-a081-2be82d7aafdf" containerName="istio-proxy"
Sep 17 17:57:54.296754 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.296689    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="093dba64-0665-40d2-a081-2be82d7aafdf" containerName="istio-proxy"
Sep 17 17:57:54.296754 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.296729    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ade6400-1078-4f31-a0d2-c136ad16c90b" containerName="main"
Sep 17 17:57:54.296754 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.296737    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ade6400-1078-4f31-a0d2-c136ad16c90b" containerName="main"
Sep 17 17:57:54.296950 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.296803    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ade6400-1078-4f31-a0d2-c136ad16c90b" containerName="main"
Sep 17 17:57:54.296950 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.296818    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="093dba64-0665-40d2-a081-2be82d7aafdf" containerName="istio-proxy"
Sep 17 17:57:54.300885 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.300864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.304651 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.304629    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-s59q6\""
Sep 17 17:57:54.304771 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.304631    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 17:57:54.304771 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.304638    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 17:57:54.305064 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.305050    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 17:57:54.319135 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.319110    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"]
Sep 17 17:57:54.349847 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.349825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.349939 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.349872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rct\" (UniqueName: \"kubernetes.io/projected/291f57fc-c600-4114-868a-0364349048d3-kube-api-access-n8rct\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.349939 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.349895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291f57fc-c600-4114-868a-0364349048d3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.350021 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.349972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.350021 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.350006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.350088 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.350053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.450646 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.450613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rct\" (UniqueName: \"kubernetes.io/projected/291f57fc-c600-4114-868a-0364349048d3-kube-api-access-n8rct\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.450841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.450652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291f57fc-c600-4114-868a-0364349048d3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.450841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.450678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.450841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.450710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.450841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.450750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.450841 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.450784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.451117 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.451081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.451182 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.451167    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.451240 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.451209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.453081 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.453059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.453332 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.453315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291f57fc-c600-4114-868a-0364349048d3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.459126 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.459104    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rct\" (UniqueName: \"kubernetes.io/projected/291f57fc-c600-4114-868a-0364349048d3-kube-api-access-n8rct\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.610591 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.610494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:54.882439 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:54.882341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"]
Sep 17 17:57:54.885431 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:57:54.885387    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291f57fc_c600_4114_868a_0364349048d3.slice/crio-d52b30705d79edc2302b22f013390e84d05ea672b4b5c902690e98ad5cef254e WatchSource:0}: Error finding container d52b30705d79edc2302b22f013390e84d05ea672b4b5c902690e98ad5cef254e: Status 404 returned error can't find the container with id d52b30705d79edc2302b22f013390e84d05ea672b4b5c902690e98ad5cef254e
Sep 17 17:57:55.105667 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:55.105628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh" event={"ID":"291f57fc-c600-4114-868a-0364349048d3","Type":"ContainerStarted","Data":"1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918"}
Sep 17 17:57:55.105824 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:55.105681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh" event={"ID":"291f57fc-c600-4114-868a-0364349048d3","Type":"ContainerStarted","Data":"d52b30705d79edc2302b22f013390e84d05ea672b4b5c902690e98ad5cef254e"}
Sep 17 17:57:55.123247 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:55.123204    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh" podStartSLOduration=1.1231868839999999 podStartE2EDuration="1.123186884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:55.122185811 +0000 UTC m=+1017.957955045" watchObservedRunningTime="2026-09-17 17:57:55.123186884 +0000 UTC m=+1017.958956141"
Sep 17 17:57:55.611057 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:55.610974    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:59.113696 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.113665    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"]
Sep 17 17:57:59.122205 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:57:59.122166    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:57:59.122205 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:59.122205 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:59.122205 ip-10-0-129-186 kubenswrapper[2565]:  > containerID="1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918" cmd=["/bin/sleep","15"]
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.122215    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh" message="" err=<
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]:  >
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:57:59.122226    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh" podUID="291f57fc-c600-4114-868a-0364349048d3" containerName="main" containerID="cri-o://1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918"
Sep 17 17:57:59.122757 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.122270    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh" podUID="291f57fc-c600-4114-868a-0364349048d3" containerName="main" containerID="cri-o://1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918" gracePeriod=60
Sep 17 17:57:59.457576 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.457553    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:57:59.490363 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490340    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-model-cache\") pod \"291f57fc-c600-4114-868a-0364349048d3\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") "
Sep 17 17:57:59.490538 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490385    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291f57fc-c600-4114-868a-0364349048d3-tls-certs\") pod \"291f57fc-c600-4114-868a-0364349048d3\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") "
Sep 17 17:57:59.490538 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490428    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8rct\" (UniqueName: \"kubernetes.io/projected/291f57fc-c600-4114-868a-0364349048d3-kube-api-access-n8rct\") pod \"291f57fc-c600-4114-868a-0364349048d3\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") "
Sep 17 17:57:59.490538 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490502    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-dshm\") pod \"291f57fc-c600-4114-868a-0364349048d3\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") "
Sep 17 17:57:59.490538 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-home\") pod \"291f57fc-c600-4114-868a-0364349048d3\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") "
Sep 17 17:57:59.490817 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-tmp-dir\") pod \"291f57fc-c600-4114-868a-0364349048d3\" (UID: \"291f57fc-c600-4114-868a-0364349048d3\") "
Sep 17 17:57:59.490817 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "291f57fc-c600-4114-868a-0364349048d3" (UID: "291f57fc-c600-4114-868a-0364349048d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:59.490817 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490764    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:59.490972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-home" (OuterVolumeSpecName: "home") pod "291f57fc-c600-4114-868a-0364349048d3" (UID: "291f57fc-c600-4114-868a-0364349048d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:59.490972 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.490866    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "291f57fc-c600-4114-868a-0364349048d3" (UID: "291f57fc-c600-4114-868a-0364349048d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:59.492756 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.492732    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-dshm" (OuterVolumeSpecName: "dshm") pod "291f57fc-c600-4114-868a-0364349048d3" (UID: "291f57fc-c600-4114-868a-0364349048d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:59.492756 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.492731    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/291f57fc-c600-4114-868a-0364349048d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "291f57fc-c600-4114-868a-0364349048d3" (UID: "291f57fc-c600-4114-868a-0364349048d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:59.492944 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.492922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291f57fc-c600-4114-868a-0364349048d3-kube-api-access-n8rct" (OuterVolumeSpecName: "kube-api-access-n8rct") pod "291f57fc-c600-4114-868a-0364349048d3" (UID: "291f57fc-c600-4114-868a-0364349048d3"). InnerVolumeSpecName "kube-api-access-n8rct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:59.591638 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.591611    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rct\" (UniqueName: \"kubernetes.io/projected/291f57fc-c600-4114-868a-0364349048d3-kube-api-access-n8rct\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:59.591638 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.591634    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:59.591638 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.591644    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:59.591845 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.591651    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291f57fc-c600-4114-868a-0364349048d3-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:57:59.591845 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:57:59.591660    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291f57fc-c600-4114-868a-0364349048d3-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:58:00.125113 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.125080    2565 generic.go:356] "Generic (PLEG): container finished" podID="291f57fc-c600-4114-868a-0364349048d3" containerID="1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918" exitCode=0
Sep 17 17:58:00.125616 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.125122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh" event={"ID":"291f57fc-c600-4114-868a-0364349048d3","Type":"ContainerDied","Data":"1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918"}
Sep 17 17:58:00.125616 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.125133    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"
Sep 17 17:58:00.125616 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.125164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh" event={"ID":"291f57fc-c600-4114-868a-0364349048d3","Type":"ContainerDied","Data":"d52b30705d79edc2302b22f013390e84d05ea672b4b5c902690e98ad5cef254e"}
Sep 17 17:58:00.125616 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.125184    2565 scope.go:117] "RemoveContainer" containerID="1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918"
Sep 17 17:58:00.133773 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.133757    2565 scope.go:117] "RemoveContainer" containerID="1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918"
Sep 17 17:58:00.134026 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:58:00.134006    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918\": container with ID starting with 1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918 not found: ID does not exist" containerID="1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918"
Sep 17 17:58:00.134069 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.134034    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918"} err="failed to get container status \"1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918\": rpc error: code = NotFound desc = could not find container \"1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918\": container with ID starting with 1070e4f56518251370c3885abb22e0a058046c13404f15dfd2e991dd8a0ed918 not found: ID does not exist"
Sep 17 17:58:00.147518 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.147493    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"]
Sep 17 17:58:00.149335 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:00.149307    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf7wkwh"]
Sep 17 17:58:01.348720 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:01.348688    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291f57fc-c600-4114-868a-0364349048d3" path="/var/lib/kubelet/pods/291f57fc-c600-4114-868a-0364349048d3/volumes"
Sep 17 17:58:11.802786 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.802749    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"]
Sep 17 17:58:11.803148 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.803041    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291f57fc-c600-4114-868a-0364349048d3" containerName="main"
Sep 17 17:58:11.803148 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.803053    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="291f57fc-c600-4114-868a-0364349048d3" containerName="main"
Sep 17 17:58:11.803148 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.803111    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="291f57fc-c600-4114-868a-0364349048d3" containerName="main"
Sep 17 17:58:11.806117 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.806097    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.808335 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.808312    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-6wwp8\""
Sep 17 17:58:11.808540 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.808315    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:58:11.808540 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.808317    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 17:58:11.808685 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.808639    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 17:58:11.808751 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.808645    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 17:58:11.808804 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.808762    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 17:58:11.821187 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.821165    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"]
Sep 17 17:58:11.886121 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.886095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tls-certs\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.886253 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.886140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-model-cache\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.886253 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.886158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-cabundle-cert\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.886253 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.886200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-dshm\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.886253 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.886231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tmp-dir\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.886385 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.886305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2456c\" (UniqueName: \"kubernetes.io/projected/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kube-api-access-2456c\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.886385 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.886346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kserve-provision-location\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.886385 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.886371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-home\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.987690 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.987652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kserve-provision-location\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.987884 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.987703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-home\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.987884 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.987821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tls-certs\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988003 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.987902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-model-cache\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988003 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.987931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-cabundle-cert\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988003 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.987962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-dshm\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988003 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.987995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tmp-dir\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988197 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.988042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2456c\" (UniqueName: \"kubernetes.io/projected/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kube-api-access-2456c\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988197 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.988091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-home\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988197 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.988115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kserve-provision-location\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988420 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.988374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-model-cache\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988420 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.988388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tmp-dir\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.988657 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.988640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-cabundle-cert\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.990176 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.990157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-dshm\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.990334 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.990315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tls-certs\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:11.996431 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:11.996381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2456c\" (UniqueName: \"kubernetes.io/projected/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kube-api-access-2456c\") pod \"auth-enabled-test-kserve-5994fd84df-wcrbk\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:12.062520 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.062431    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"]
Sep 17 17:58:12.065243 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.065227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:12.067174 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.067153    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-8d5l2\""
Sep 17 17:58:12.075339 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.075318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"]
Sep 17 17:58:12.116058 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.116030    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:12.189699 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.189672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw\" (UID: \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:12.189868 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.189732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpprj\" (UniqueName: \"kubernetes.io/projected/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-kube-api-access-lpprj\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw\" (UID: \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:12.290687 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.290650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw\" (UID: \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:12.290865 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.290715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpprj\" (UniqueName: \"kubernetes.io/projected/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-kube-api-access-lpprj\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw\" (UID: \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:12.293041 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.293018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw\" (UID: \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:12.300795 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.300775    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpprj\" (UniqueName: \"kubernetes.io/projected/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-kube-api-access-lpprj\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw\" (UID: \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:12.375224 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.375160    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:12.390319 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.390292    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"]
Sep 17 17:58:12.394067 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:58:12.394042    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dbdba0_40ad_4626_b8e8_c3de8f7614bc.slice/crio-b2e9e8d9d16b35ba2191d164d8466a99b68a5e8ba76d9cb4faaa8cb231dd9493 WatchSource:0}: Error finding container b2e9e8d9d16b35ba2191d164d8466a99b68a5e8ba76d9cb4faaa8cb231dd9493: Status 404 returned error can't find the container with id b2e9e8d9d16b35ba2191d164d8466a99b68a5e8ba76d9cb4faaa8cb231dd9493
Sep 17 17:58:12.646760 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:12.646682    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"]
Sep 17 17:58:12.649701 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:58:12.649672    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5ccd4e_ab4e_4640_a13a_55570f1be1c1.slice/crio-0319f7ed087c4eccc60998a6891bf5fb5cb52267c20cb9c7b3beb935804efd87 WatchSource:0}: Error finding container 0319f7ed087c4eccc60998a6891bf5fb5cb52267c20cb9c7b3beb935804efd87: Status 404 returned error can't find the container with id 0319f7ed087c4eccc60998a6891bf5fb5cb52267c20cb9c7b3beb935804efd87
Sep 17 17:58:13.179702 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:13.179661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" event={"ID":"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1","Type":"ContainerStarted","Data":"fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234"}
Sep 17 17:58:13.180169 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:13.179729    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" event={"ID":"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1","Type":"ContainerStarted","Data":"0319f7ed087c4eccc60998a6891bf5fb5cb52267c20cb9c7b3beb935804efd87"}
Sep 17 17:58:13.180169 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:13.180092    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:13.181884 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:13.181853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" event={"ID":"31dbdba0-40ad-4626-b8e8-c3de8f7614bc","Type":"ContainerStarted","Data":"b2e9e8d9d16b35ba2191d164d8466a99b68a5e8ba76d9cb4faaa8cb231dd9493"}
Sep 17 17:58:13.199511 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:13.199459    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" podStartSLOduration=1.199439324 podStartE2EDuration="1.199439324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:13.198757835 +0000 UTC m=+1036.034527082" watchObservedRunningTime="2026-09-17 17:58:13.199439324 +0000 UTC m=+1036.035208549"
Sep 17 17:58:16.196411 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:16.196359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" event={"ID":"31dbdba0-40ad-4626-b8e8-c3de8f7614bc","Type":"ContainerStarted","Data":"339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a"}
Sep 17 17:58:18.205781 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:18.205749    2565 generic.go:356] "Generic (PLEG): container finished" podID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerID="339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a" exitCode=0
Sep 17 17:58:18.206134 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:18.205819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" event={"ID":"31dbdba0-40ad-4626-b8e8-c3de8f7614bc","Type":"ContainerDied","Data":"339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a"}
Sep 17 17:58:44.188585 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:44.188507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 17:58:46.748081 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:46.747987    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"]
Sep 17 17:58:47.350987 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:47.350956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" event={"ID":"31dbdba0-40ad-4626-b8e8-c3de8f7614bc","Type":"ContainerStarted","Data":"00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0"}
Sep 17 17:58:47.375808 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:47.375762    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podStartSLOduration=2.5263323 podStartE2EDuration="36.375746957s" totalImagesPullingTime="31.782961126s" totalInitContainerRuntime="2.066453531s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:47.375532493 +0000 UTC m=+1070.211301716" watchObservedRunningTime="2026-09-17 17:58:47.375746957 +0000 UTC m=+1070.211516179"
Sep 17 17:58:48.116386 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:48.116351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:52.116872 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:52.116833    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 17:58:52.118443 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:52.118387    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 17:58:53.887482 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:58:53.887446    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:01.768956 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:01.768913    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerName="main" containerID="cri-o://992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639" gracePeriod=45
Sep 17 17:59:02.116551 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:02.116460    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 17:59:03.887655 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:03.887615    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:12.117465 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:12.117419    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 17:59:13.887509 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:13.887469    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:13.887946 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:13.887588    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:59:22.117017 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:22.116976    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 17:59:23.888054 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:23.888015    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:32.009834 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.009804    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:59:32.071887 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.071796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz4z4\" (UniqueName: \"kubernetes.io/projected/001270b0-71a2-4ef8-8c16-fe0bef7702a3-kube-api-access-rz4z4\") pod \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\" (UID: \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\") "
Sep 17 17:59:32.072041 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.071917    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001270b0-71a2-4ef8-8c16-fe0bef7702a3-tls-certs\") pod \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\" (UID: \"001270b0-71a2-4ef8-8c16-fe0bef7702a3\") "
Sep 17 17:59:32.074046 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.074012    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001270b0-71a2-4ef8-8c16-fe0bef7702a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "001270b0-71a2-4ef8-8c16-fe0bef7702a3" (UID: "001270b0-71a2-4ef8-8c16-fe0bef7702a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:32.074151 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.074052    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001270b0-71a2-4ef8-8c16-fe0bef7702a3-kube-api-access-rz4z4" (OuterVolumeSpecName: "kube-api-access-rz4z4") pod "001270b0-71a2-4ef8-8c16-fe0bef7702a3" (UID: "001270b0-71a2-4ef8-8c16-fe0bef7702a3"). InnerVolumeSpecName "kube-api-access-rz4z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:32.116445 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.116390    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 17:59:32.173198 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.173170    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001270b0-71a2-4ef8-8c16-fe0bef7702a3-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:59:32.173198 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.173197    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4z4\" (UniqueName: \"kubernetes.io/projected/001270b0-71a2-4ef8-8c16-fe0bef7702a3-kube-api-access-rz4z4\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 17:59:32.526436 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.526403    2565 generic.go:356] "Generic (PLEG): container finished" podID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerID="992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639" exitCode=0
Sep 17 17:59:32.526614 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.526446    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"
Sep 17 17:59:32.526614 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.526457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" event={"ID":"001270b0-71a2-4ef8-8c16-fe0bef7702a3","Type":"ContainerDied","Data":"992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639"}
Sep 17 17:59:32.526614 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.526512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl" event={"ID":"001270b0-71a2-4ef8-8c16-fe0bef7702a3","Type":"ContainerDied","Data":"8b82d36760cf6579608330495fb4365cec4e1d28d2f73f4a0b2fa75f81ffac51"}
Sep 17 17:59:32.526614 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.526538    2565 scope.go:117] "RemoveContainer" containerID="992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639"
Sep 17 17:59:32.536099 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.536082    2565 scope.go:117] "RemoveContainer" containerID="992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639"
Sep 17 17:59:32.536369 ip-10-0-129-186 kubenswrapper[2565]: E0917 17:59:32.536350    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639\": container with ID starting with 992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639 not found: ID does not exist" containerID="992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639"
Sep 17 17:59:32.536459 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.536378    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639"} err="failed to get container status \"992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639\": rpc error: code = NotFound desc = could not find container \"992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639\": container with ID starting with 992f3a7533893223eebc351a5413fc20d2067af12660e02b6a76d1aa70a20639 not found: ID does not exist"
Sep 17 17:59:32.547431 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.547409    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"]
Sep 17 17:59:32.549281 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:32.549262    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b6xnl"]
Sep 17 17:59:33.349439 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:33.349387    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" path="/var/lib/kubelet/pods/001270b0-71a2-4ef8-8c16-fe0bef7702a3/volumes"
Sep 17 17:59:42.117247 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:42.117201    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 17:59:44.269074 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.269037    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"]
Sep 17 17:59:44.269450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.269333    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerName="main"
Sep 17 17:59:44.269450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.269344    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerName="main"
Sep 17 17:59:44.269450 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.269408    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="001270b0-71a2-4ef8-8c16-fe0bef7702a3" containerName="main"
Sep 17 17:59:44.271886 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.271862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:44.274095 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.274074    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 17:59:44.274763 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.274742    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-f8vzn\""
Sep 17 17:59:44.274876 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.274763    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-2c2zp\""
Sep 17 17:59:44.274876 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.274828    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 17:59:44.274876 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.274864    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 17:59:44.285638 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.285617    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"]
Sep 17 17:59:44.348858 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.348828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f250f27-7b84-4f7b-a691-586c45460b68-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc\" (UID: \"5f250f27-7b84-4f7b-a691-586c45460b68\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:44.349030 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.348868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9mcj\" (UniqueName: \"kubernetes.io/projected/5f250f27-7b84-4f7b-a691-586c45460b68-kube-api-access-m9mcj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc\" (UID: \"5f250f27-7b84-4f7b-a691-586c45460b68\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:44.450076 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.450041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f250f27-7b84-4f7b-a691-586c45460b68-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc\" (UID: \"5f250f27-7b84-4f7b-a691-586c45460b68\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:44.450076 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.450082    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9mcj\" (UniqueName: \"kubernetes.io/projected/5f250f27-7b84-4f7b-a691-586c45460b68-kube-api-access-m9mcj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc\" (UID: \"5f250f27-7b84-4f7b-a691-586c45460b68\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:44.452804 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.452773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f250f27-7b84-4f7b-a691-586c45460b68-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc\" (UID: \"5f250f27-7b84-4f7b-a691-586c45460b68\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:44.458897 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.458862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9mcj\" (UniqueName: \"kubernetes.io/projected/5f250f27-7b84-4f7b-a691-586c45460b68-kube-api-access-m9mcj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc\" (UID: \"5f250f27-7b84-4f7b-a691-586c45460b68\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:44.584843 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.584751    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:44.871264 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:44.871175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"]
Sep 17 17:59:44.874671 ip-10-0-129-186 kubenswrapper[2565]: W0917 17:59:44.874642    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f250f27_7b84_4f7b_a691_586c45460b68.slice/crio-56e570218bd42350c3b968017802ea1ae0dcb933665aaf9385c839312dd3b43b WatchSource:0}: Error finding container 56e570218bd42350c3b968017802ea1ae0dcb933665aaf9385c839312dd3b43b: Status 404 returned error can't find the container with id 56e570218bd42350c3b968017802ea1ae0dcb933665aaf9385c839312dd3b43b
Sep 17 17:59:45.593119 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:45.593083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" event={"ID":"5f250f27-7b84-4f7b-a691-586c45460b68","Type":"ContainerStarted","Data":"d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3"}
Sep 17 17:59:45.593516 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:45.593145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" event={"ID":"5f250f27-7b84-4f7b-a691-586c45460b68","Type":"ContainerStarted","Data":"56e570218bd42350c3b968017802ea1ae0dcb933665aaf9385c839312dd3b43b"}
Sep 17 17:59:45.593516 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:45.593196    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 17:59:45.611388 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:45.611340    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" podStartSLOduration=1.611324821 podStartE2EDuration="1.611324821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:45.610443711 +0000 UTC m=+1128.446212956" watchObservedRunningTime="2026-09-17 17:59:45.611324821 +0000 UTC m=+1128.447094044"
Sep 17 17:59:52.116699 ip-10-0-129-186 kubenswrapper[2565]: I0917 17:59:52.116650    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 18:00:02.116930 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:02.116886    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 18:00:12.116951 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:12.116858    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 18:00:16.599915 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:16.599884    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 18:00:18.564726 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:18.564691    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"]
Sep 17 18:00:22.116774 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:22.116732    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 18:00:26.600460 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:26.600419    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:32.116829 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:32.116790    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 18:00:33.584822 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:33.584779    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" containerName="main" containerID="cri-o://d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3" gracePeriod=45
Sep 17 18:00:36.599956 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:36.599918    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:42.126665 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:42.126635    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 18:00:42.134552 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:42.134524    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 18:00:46.600128 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:46.600089    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:46.600587 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:46.600191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 18:00:46.928736 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:46.928701    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"]
Sep 17 18:00:46.932687 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:46.932647    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"]
Sep 17 18:00:54.189000 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:54.188961    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:56.599963 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:56.599928    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:59.326858 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:59.326832    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:00:59.327724 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:00:59.327702    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:01:01.955613 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:01.955576    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" containerID="cri-o://00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0" gracePeriod=45
Sep 17 18:01:01.956947 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:01.956912    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerName="main" containerID="cri-o://fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234" gracePeriod=45
Sep 17 18:01:02.127734 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.127696    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 18:01:02.700455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.700425    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 18:01:02.796950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.796880    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2456c\" (UniqueName: \"kubernetes.io/projected/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kube-api-access-2456c\") pod \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") "
Sep 17 18:01:02.796950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.796925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tmp-dir\") pod \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") "
Sep 17 18:01:02.796950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.796952    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-model-cache\") pod \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") "
Sep 17 18:01:02.797222 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.796972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kserve-provision-location\") pod \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") "
Sep 17 18:01:02.797222 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.796996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tls-certs\") pod \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") "
Sep 17 18:01:02.797222 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.797011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-dshm\") pod \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") "
Sep 17 18:01:02.797222 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.797054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-cabundle-cert\") pod \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") "
Sep 17 18:01:02.797222 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.797091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-home\") pod \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\" (UID: \"31dbdba0-40ad-4626-b8e8-c3de8f7614bc\") "
Sep 17 18:01:02.797450 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.797259    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "31dbdba0-40ad-4626-b8e8-c3de8f7614bc" (UID: "31dbdba0-40ad-4626-b8e8-c3de8f7614bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:02.797928 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.797890    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31dbdba0-40ad-4626-b8e8-c3de8f7614bc" (UID: "31dbdba0-40ad-4626-b8e8-c3de8f7614bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:02.798077 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.798053    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-home" (OuterVolumeSpecName: "home") pod "31dbdba0-40ad-4626-b8e8-c3de8f7614bc" (UID: "31dbdba0-40ad-4626-b8e8-c3de8f7614bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:02.799278 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.799254    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kube-api-access-2456c" (OuterVolumeSpecName: "kube-api-access-2456c") pod "31dbdba0-40ad-4626-b8e8-c3de8f7614bc" (UID: "31dbdba0-40ad-4626-b8e8-c3de8f7614bc"). InnerVolumeSpecName "kube-api-access-2456c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:02.799695 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.799674    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31dbdba0-40ad-4626-b8e8-c3de8f7614bc" (UID: "31dbdba0-40ad-4626-b8e8-c3de8f7614bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:02.799787 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.799706    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-dshm" (OuterVolumeSpecName: "dshm") pod "31dbdba0-40ad-4626-b8e8-c3de8f7614bc" (UID: "31dbdba0-40ad-4626-b8e8-c3de8f7614bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:02.818458 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.818377    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31dbdba0-40ad-4626-b8e8-c3de8f7614bc" (UID: "31dbdba0-40ad-4626-b8e8-c3de8f7614bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:02.824734 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.824710    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31dbdba0-40ad-4626-b8e8-c3de8f7614bc" (UID: "31dbdba0-40ad-4626-b8e8-c3de8f7614bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:02.897842 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.897815    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:02.897842 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.897840    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:02.897988 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.897850    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:02.897988 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.897859    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:02.897988 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.897868    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2456c\" (UniqueName: \"kubernetes.io/projected/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kube-api-access-2456c\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:02.897988 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.897877    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:02.897988 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.897887    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:02.897988 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.897895    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dbdba0-40ad-4626-b8e8-c3de8f7614bc-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:02.902293 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.902263    2565 generic.go:356] "Generic (PLEG): container finished" podID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerID="00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0" exitCode=0
Sep 17 18:01:02.902421 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.902317    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"
Sep 17 18:01:02.902421 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.902325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" event={"ID":"31dbdba0-40ad-4626-b8e8-c3de8f7614bc","Type":"ContainerDied","Data":"00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0"}
Sep 17 18:01:02.902421 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.902378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk" event={"ID":"31dbdba0-40ad-4626-b8e8-c3de8f7614bc","Type":"ContainerDied","Data":"b2e9e8d9d16b35ba2191d164d8466a99b68a5e8ba76d9cb4faaa8cb231dd9493"}
Sep 17 18:01:02.902421 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.902421    2565 scope.go:117] "RemoveContainer" containerID="00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0"
Sep 17 18:01:02.911256 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.911241    2565 scope.go:117] "RemoveContainer" containerID="339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a"
Sep 17 18:01:02.918815 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.918799    2565 scope.go:117] "RemoveContainer" containerID="00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0"
Sep 17 18:01:02.919083 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:01:02.919063    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0\": container with ID starting with 00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0 not found: ID does not exist" containerID="00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0"
Sep 17 18:01:02.919127 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.919092    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0"} err="failed to get container status \"00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0\": rpc error: code = NotFound desc = could not find container \"00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0\": container with ID starting with 00c2e2b66928be22d7dfefa2a9e5fb7977356667f8cd6a293ac7cba75d5129d0 not found: ID does not exist"
Sep 17 18:01:02.919127 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.919109    2565 scope.go:117] "RemoveContainer" containerID="339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a"
Sep 17 18:01:02.919361 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:01:02.919332    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a\": container with ID starting with 339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a not found: ID does not exist" containerID="339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a"
Sep 17 18:01:02.919472 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.919360    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a"} err="failed to get container status \"339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a\": rpc error: code = NotFound desc = could not find container \"339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a\": container with ID starting with 339ccd9f8a55ad0c2ba4401050c19de48caedf9071fea5cf19e663326b9b0b8a not found: ID does not exist"
Sep 17 18:01:02.923112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.923088    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"]
Sep 17 18:01:02.926727 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:02.926707    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5994fd84df-wcrbk"]
Sep 17 18:01:03.348636 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.348605    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" path="/var/lib/kubelet/pods/31dbdba0-40ad-4626-b8e8-c3de8f7614bc/volumes"
Sep 17 18:01:03.832310 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.832283    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 18:01:03.906901 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.906816    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f250f27-7b84-4f7b-a691-586c45460b68-tls-certs\") pod \"5f250f27-7b84-4f7b-a691-586c45460b68\" (UID: \"5f250f27-7b84-4f7b-a691-586c45460b68\") "
Sep 17 18:01:03.907040 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.906915    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9mcj\" (UniqueName: \"kubernetes.io/projected/5f250f27-7b84-4f7b-a691-586c45460b68-kube-api-access-m9mcj\") pod \"5f250f27-7b84-4f7b-a691-586c45460b68\" (UID: \"5f250f27-7b84-4f7b-a691-586c45460b68\") "
Sep 17 18:01:03.912281 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.912248    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f250f27-7b84-4f7b-a691-586c45460b68-kube-api-access-m9mcj" (OuterVolumeSpecName: "kube-api-access-m9mcj") pod "5f250f27-7b84-4f7b-a691-586c45460b68" (UID: "5f250f27-7b84-4f7b-a691-586c45460b68"). InnerVolumeSpecName "kube-api-access-m9mcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:03.912455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.912286    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f250f27-7b84-4f7b-a691-586c45460b68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f250f27-7b84-4f7b-a691-586c45460b68" (UID: "5f250f27-7b84-4f7b-a691-586c45460b68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:03.914755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.914722    2565 generic.go:356] "Generic (PLEG): container finished" podID="5f250f27-7b84-4f7b-a691-586c45460b68" containerID="d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3" exitCode=0
Sep 17 18:01:03.914861 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.914779    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"
Sep 17 18:01:03.914861 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.914804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" event={"ID":"5f250f27-7b84-4f7b-a691-586c45460b68","Type":"ContainerDied","Data":"d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3"}
Sep 17 18:01:03.914954 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.914893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc" event={"ID":"5f250f27-7b84-4f7b-a691-586c45460b68","Type":"ContainerDied","Data":"56e570218bd42350c3b968017802ea1ae0dcb933665aaf9385c839312dd3b43b"}
Sep 17 18:01:03.914954 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.914926    2565 scope.go:117] "RemoveContainer" containerID="d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3"
Sep 17 18:01:03.925280 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.925263    2565 scope.go:117] "RemoveContainer" containerID="d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3"
Sep 17 18:01:03.925561 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:01:03.925538    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3\": container with ID starting with d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3 not found: ID does not exist" containerID="d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3"
Sep 17 18:01:03.925639 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.925567    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3"} err="failed to get container status \"d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3\": rpc error: code = NotFound desc = could not find container \"d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3\": container with ID starting with d9f7876c0c4843f8bea44e24cfbf2dc783c3b0c10212e3dc606ee9f10b7009e3 not found: ID does not exist"
Sep 17 18:01:03.935787 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.935769    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"]
Sep 17 18:01:03.940261 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:03.940241    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfw8cc"]
Sep 17 18:01:04.007728 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:04.007687    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f250f27-7b84-4f7b-a691-586c45460b68-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:04.007728 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:04.007724    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9mcj\" (UniqueName: \"kubernetes.io/projected/5f250f27-7b84-4f7b-a691-586c45460b68-kube-api-access-m9mcj\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:04.188894 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:04.188853    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:01:05.348792 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:05.348761    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" path="/var/lib/kubelet/pods/5f250f27-7b84-4f7b-a691-586c45460b68/volumes"
Sep 17 18:01:13.400915 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.400885    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"]
Sep 17 18:01:13.401264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.401133    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" containerName="main"
Sep 17 18:01:13.401264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.401144    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" containerName="main"
Sep 17 18:01:13.401264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.401156    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main"
Sep 17 18:01:13.401264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.401161    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main"
Sep 17 18:01:13.401264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.401170    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="storage-initializer"
Sep 17 18:01:13.401264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.401176    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="storage-initializer"
Sep 17 18:01:13.401264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.401223    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f250f27-7b84-4f7b-a691-586c45460b68" containerName="main"
Sep 17 18:01:13.401264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.401230    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="31dbdba0-40ad-4626-b8e8-c3de8f7614bc" containerName="main"
Sep 17 18:01:13.405571 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.405555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.408153 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.408133    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:13.408814 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.408788    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 18:01:13.408944 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.408844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:13.408944 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.408857    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 18:01:13.408944 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.408844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 18:01:13.408944 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.408901    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-qbz5s\""
Sep 17 18:01:13.418761 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.418742    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"]
Sep 17 18:01:13.482437 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.482412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-dshm\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.482549 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.482444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-home\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.482549 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.482475    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-model-cache\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.482549 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.482504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f363c646-16d3-4aff-ac09-9178b54501cd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.482647 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.482575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f363c646-16d3-4aff-ac09-9178b54501cd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.482647 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.482602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.482647 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.482631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.482740 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.482661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hls\" (UniqueName: \"kubernetes.io/projected/f363c646-16d3-4aff-ac09-9178b54501cd-kube-api-access-w6hls\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.583651 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.583607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.583651 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.583654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hls\" (UniqueName: \"kubernetes.io/projected/f363c646-16d3-4aff-ac09-9178b54501cd-kube-api-access-w6hls\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.583848 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.583677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-dshm\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.583848 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.583798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-home\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.583909 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.583865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-model-cache\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.583909 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.583892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f363c646-16d3-4aff-ac09-9178b54501cd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.583986 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.583948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f363c646-16d3-4aff-ac09-9178b54501cd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.584040 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.583988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.584098 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.584061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.584163 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.584143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-home\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.584268 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.584248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-model-cache\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.584330 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.584249    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.584632 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.584611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f363c646-16d3-4aff-ac09-9178b54501cd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.585899 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.585882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-dshm\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.586235 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.586218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f363c646-16d3-4aff-ac09-9178b54501cd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.594048 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.594025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hls\" (UniqueName: \"kubernetes.io/projected/f363c646-16d3-4aff-ac09-9178b54501cd-kube-api-access-w6hls\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-ctsmw\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.648469 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.648439    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"]
Sep 17 18:01:13.652670 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.652617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:13.654897 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.654879    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-msqwz\""
Sep 17 18:01:13.662381 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.662357    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"]
Sep 17 18:01:13.715765 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.715741    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:13.786072 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.786034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7d8\" (UniqueName: \"kubernetes.io/projected/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-kube-api-access-6r7d8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l\" (UID: \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:13.786212 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.786187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l\" (UID: \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:13.887567 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.887527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l\" (UID: \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:13.887722 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.887584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7d8\" (UniqueName: \"kubernetes.io/projected/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-kube-api-access-6r7d8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l\" (UID: \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:13.889898 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.889881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l\" (UID: \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:13.896532 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.896514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7d8\" (UniqueName: \"kubernetes.io/projected/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-kube-api-access-6r7d8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l\" (UID: \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:13.964425 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.964380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:13.989315 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:13.989294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"]
Sep 17 18:01:13.991691 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:01:13.991667    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf363c646_16d3_4aff_ac09_9178b54501cd.slice/crio-8a03fc2e4d9907f7dd09fb35881b62615bae6b3113267ff13e7d6d59ae762fc3 WatchSource:0}: Error finding container 8a03fc2e4d9907f7dd09fb35881b62615bae6b3113267ff13e7d6d59ae762fc3: Status 404 returned error can't find the container with id 8a03fc2e4d9907f7dd09fb35881b62615bae6b3113267ff13e7d6d59ae762fc3
Sep 17 18:01:14.188965 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:14.188922    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:01:14.189138 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:14.189049    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 18:01:14.280423 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:14.280323    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"]
Sep 17 18:01:14.284648 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:01:14.284620    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b61d7e_0bcf_45d4_887e_4472ab8f0aac.slice/crio-a9ee896e6232af42a6a240d686f7040e4547721f42428ac944511c4476821b09 WatchSource:0}: Error finding container a9ee896e6232af42a6a240d686f7040e4547721f42428ac944511c4476821b09: Status 404 returned error can't find the container with id a9ee896e6232af42a6a240d686f7040e4547721f42428ac944511c4476821b09
Sep 17 18:01:14.958309 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:14.958272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" event={"ID":"f363c646-16d3-4aff-ac09-9178b54501cd","Type":"ContainerStarted","Data":"ab60d694648e9aee0613aea33083ef95cb4402d8e0df0fd38187288c345dc870"}
Sep 17 18:01:14.958967 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:14.958926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" event={"ID":"f363c646-16d3-4aff-ac09-9178b54501cd","Type":"ContainerStarted","Data":"8a03fc2e4d9907f7dd09fb35881b62615bae6b3113267ff13e7d6d59ae762fc3"}
Sep 17 18:01:14.960084 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:14.960060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" event={"ID":"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac","Type":"ContainerStarted","Data":"627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899"}
Sep 17 18:01:14.960196 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:14.960106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" event={"ID":"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac","Type":"ContainerStarted","Data":"a9ee896e6232af42a6a240d686f7040e4547721f42428ac944511c4476821b09"}
Sep 17 18:01:14.960196 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:14.960143    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:15.000351 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:15.000310    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" podStartSLOduration=2.00029583 podStartE2EDuration="2.00029583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:14.999486563 +0000 UTC m=+1217.835255800" watchObservedRunningTime="2026-09-17 18:01:15.00029583 +0000 UTC m=+1217.836065053"
Sep 17 18:01:16.970005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:16.969974    2565 generic.go:356] "Generic (PLEG): container finished" podID="f363c646-16d3-4aff-ac09-9178b54501cd" containerID="ab60d694648e9aee0613aea33083ef95cb4402d8e0df0fd38187288c345dc870" exitCode=0
Sep 17 18:01:16.970381 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:16.970023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" event={"ID":"f363c646-16d3-4aff-ac09-9178b54501cd","Type":"ContainerDied","Data":"ab60d694648e9aee0613aea33083ef95cb4402d8e0df0fd38187288c345dc870"}
Sep 17 18:01:17.975789 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:17.975758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" event={"ID":"f363c646-16d3-4aff-ac09-9178b54501cd","Type":"ContainerStarted","Data":"63e346b39eeef2fadeee4c06913f3bcbc6624072b162e84a9ea59a736ddac724"}
Sep 17 18:01:17.996498 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:17.996442    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" podStartSLOduration=2.876142119 podStartE2EDuration="4.996425258s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.120283139s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:17.995950025 +0000 UTC m=+1220.831719249" watchObservedRunningTime="2026-09-17 18:01:17.996425258 +0000 UTC m=+1220.832194491"
Sep 17 18:01:18.716497 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:18.716457    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:23.716101 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:23.716054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:23.728706 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:23.728673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:24.011958 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:24.011885    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:01:24.189056 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:24.189019    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:01:32.201687 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:32.201660    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 18:01:32.335136 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:32.335057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-tls-certs\") pod \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\" (UID: \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\") "
Sep 17 18:01:32.335252 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:32.335201    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpprj\" (UniqueName: \"kubernetes.io/projected/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-kube-api-access-lpprj\") pod \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\" (UID: \"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1\") "
Sep 17 18:01:32.337127 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:32.337099    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" (UID: "cf5ccd4e-ab4e-4640-a13a-55570f1be1c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:32.337227 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:32.337203    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-kube-api-access-lpprj" (OuterVolumeSpecName: "kube-api-access-lpprj") pod "cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" (UID: "cf5ccd4e-ab4e-4640-a13a-55570f1be1c1"). InnerVolumeSpecName "kube-api-access-lpprj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:32.436316 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:32.436277    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpprj\" (UniqueName: \"kubernetes.io/projected/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-kube-api-access-lpprj\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:32.436316 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:32.436311    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:01:33.035836 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.035799    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerID="fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234" exitCode=0
Sep 17 18:01:33.036013 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.035846    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"
Sep 17 18:01:33.036013 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.035885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" event={"ID":"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1","Type":"ContainerDied","Data":"fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234"}
Sep 17 18:01:33.036013 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.035936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw" event={"ID":"cf5ccd4e-ab4e-4640-a13a-55570f1be1c1","Type":"ContainerDied","Data":"0319f7ed087c4eccc60998a6891bf5fb5cb52267c20cb9c7b3beb935804efd87"}
Sep 17 18:01:33.036013 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.035958    2565 scope.go:117] "RemoveContainer" containerID="fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234"
Sep 17 18:01:33.044564 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.044549    2565 scope.go:117] "RemoveContainer" containerID="fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234"
Sep 17 18:01:33.044829 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:01:33.044809    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234\": container with ID starting with fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234 not found: ID does not exist" containerID="fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234"
Sep 17 18:01:33.044874 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.044836    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234"} err="failed to get container status \"fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234\": rpc error: code = NotFound desc = could not find container \"fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234\": container with ID starting with fc497031a28476cfec8775256e851fc80c5cc110a8a3c67e7107970165886234 not found: ID does not exist"
Sep 17 18:01:33.056857 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.056830    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"]
Sep 17 18:01:33.059696 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.059674    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-frltw"]
Sep 17 18:01:33.348452 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:33.348356    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" path="/var/lib/kubelet/pods/cf5ccd4e-ab4e-4640-a13a-55570f1be1c1/volumes"
Sep 17 18:01:43.110073 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.109984    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"]
Sep 17 18:01:43.110546 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.110387    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerName="main"
Sep 17 18:01:43.110546 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.110435    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerName="main"
Sep 17 18:01:43.110546 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.110536    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5ccd4e-ab4e-4640-a13a-55570f1be1c1" containerName="main"
Sep 17 18:01:43.113113 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.113092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.115245 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.115218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 18:01:43.115369 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.115218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:43.115369 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.115281    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:43.115491 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.115469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-xflrk\""
Sep 17 18:01:43.115802 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.115785    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 18:01:43.115869 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.115818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 18:01:43.129693 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.129671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"]
Sep 17 18:01:43.222223 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.222188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-dshm\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.222223 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.222225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-home\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.222551 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.222301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.222551 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.222341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrs48\" (UniqueName: \"kubernetes.io/projected/624c2094-bf22-4925-bb42-51b18fe39697-kube-api-access-hrs48\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.222551 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.222372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624c2094-bf22-4925-bb42-51b18fe39697-tls-certs\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.222551 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.222417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-model-cache\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.222551 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.222473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624c2094-bf22-4925-bb42-51b18fe39697-cabundle-cert\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.222551 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.222514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-tmp-dir\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323141 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-home\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323333 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323333 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrs48\" (UniqueName: \"kubernetes.io/projected/624c2094-bf22-4925-bb42-51b18fe39697-kube-api-access-hrs48\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323333 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624c2094-bf22-4925-bb42-51b18fe39697-tls-certs\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323333 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-model-cache\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323333 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624c2094-bf22-4925-bb42-51b18fe39697-cabundle-cert\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323333 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-tmp-dir\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323670 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323670 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-dshm\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323670 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323660    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-home\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323821 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-model-cache\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.323860 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.323826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-tmp-dir\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.324087 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.324062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624c2094-bf22-4925-bb42-51b18fe39697-cabundle-cert\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.325601 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.325581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-dshm\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.325793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.325778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624c2094-bf22-4925-bb42-51b18fe39697-tls-certs\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.331184 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.331161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrs48\" (UniqueName: \"kubernetes.io/projected/624c2094-bf22-4925-bb42-51b18fe39697-kube-api-access-hrs48\") pod \"auth-invalid-token-test-kserve-67978d878f-hqk9x\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.423096 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.423053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:43.703959 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:43.703932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"]
Sep 17 18:01:43.706389 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:01:43.706357    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624c2094_bf22_4925_bb42_51b18fe39697.slice/crio-84e235c8f0f52d2f6b971c003de9fa656d92ddbe414fc5c9664037b72a849aec WatchSource:0}: Error finding container 84e235c8f0f52d2f6b971c003de9fa656d92ddbe414fc5c9664037b72a849aec: Status 404 returned error can't find the container with id 84e235c8f0f52d2f6b971c003de9fa656d92ddbe414fc5c9664037b72a849aec
Sep 17 18:01:44.080076 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:44.079997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" event={"ID":"624c2094-bf22-4925-bb42-51b18fe39697","Type":"ContainerStarted","Data":"c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3"}
Sep 17 18:01:44.080076 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:44.080052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" event={"ID":"624c2094-bf22-4925-bb42-51b18fe39697","Type":"ContainerStarted","Data":"84e235c8f0f52d2f6b971c003de9fa656d92ddbe414fc5c9664037b72a849aec"}
Sep 17 18:01:45.966349 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:45.966316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:01:46.089548 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:46.089516    2565 generic.go:356] "Generic (PLEG): container finished" podID="624c2094-bf22-4925-bb42-51b18fe39697" containerID="c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3" exitCode=0
Sep 17 18:01:46.089687 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:46.089587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" event={"ID":"624c2094-bf22-4925-bb42-51b18fe39697","Type":"ContainerDied","Data":"c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3"}
Sep 17 18:01:47.096193 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:47.096161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" event={"ID":"624c2094-bf22-4925-bb42-51b18fe39697","Type":"ContainerStarted","Data":"719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b"}
Sep 17 18:01:47.124705 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:47.124651    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podStartSLOduration=1.879980306 podStartE2EDuration="4.124630842s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.244650536s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:47.123282038 +0000 UTC m=+1249.959051260" watchObservedRunningTime="2026-09-17 18:01:47.124630842 +0000 UTC m=+1249.960400069"
Sep 17 18:01:47.423948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:47.423923    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:53.423590 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:53.423562    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:01:53.425119 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:01:53.425092    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:02:03.423500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:03.423452    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:02:13.424427 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:13.424371    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:02:23.423588 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:23.423545    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:02:33.423678 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:33.423635    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:02:38.874197 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:38.874165    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"]
Sep 17 18:02:38.883969 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:38.883942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"]
Sep 17 18:02:38.892252 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:02:38.892218    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:02:38.892252 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:38.892252 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:38.892252 ip-10-0-129-186 kubenswrapper[2565]:  > containerID="63e346b39eeef2fadeee4c06913f3bcbc6624072b162e84a9ea59a736ddac724" cmd=["/bin/sleep","15"]
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:38.892271    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" message="" err=<
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]:  >
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:02:38.892286    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" containerName="main" containerID="cri-o://63e346b39eeef2fadeee4c06913f3bcbc6624072b162e84a9ea59a736ddac724"
Sep 17 18:02:38.892580 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:38.892340    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" containerName="main" containerID="cri-o://63e346b39eeef2fadeee4c06913f3bcbc6624072b162e84a9ea59a736ddac724" gracePeriod=60
Sep 17 18:02:39.001467 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.001433    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 17 18:02:39.312351 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.312318    2565 generic.go:356] "Generic (PLEG): container finished" podID="f363c646-16d3-4aff-ac09-9178b54501cd" containerID="63e346b39eeef2fadeee4c06913f3bcbc6624072b162e84a9ea59a736ddac724" exitCode=0
Sep 17 18:02:39.312525 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.312407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" event={"ID":"f363c646-16d3-4aff-ac09-9178b54501cd","Type":"ContainerDied","Data":"63e346b39eeef2fadeee4c06913f3bcbc6624072b162e84a9ea59a736ddac724"}
Sep 17 18:02:39.657848 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.657819    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:02:39.720601 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.720573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-dshm\") pod \"f363c646-16d3-4aff-ac09-9178b54501cd\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") "
Sep 17 18:02:39.720768 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.720610    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-kserve-provision-location\") pod \"f363c646-16d3-4aff-ac09-9178b54501cd\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") "
Sep 17 18:02:39.720768 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.720667    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f363c646-16d3-4aff-ac09-9178b54501cd-cabundle-cert\") pod \"f363c646-16d3-4aff-ac09-9178b54501cd\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") "
Sep 17 18:02:39.720768 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.720701    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6hls\" (UniqueName: \"kubernetes.io/projected/f363c646-16d3-4aff-ac09-9178b54501cd-kube-api-access-w6hls\") pod \"f363c646-16d3-4aff-ac09-9178b54501cd\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") "
Sep 17 18:02:39.720768 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.720719    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-model-cache\") pod \"f363c646-16d3-4aff-ac09-9178b54501cd\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") "
Sep 17 18:02:39.720768 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.720739    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-home\") pod \"f363c646-16d3-4aff-ac09-9178b54501cd\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") "
Sep 17 18:02:39.721032 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.720792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f363c646-16d3-4aff-ac09-9178b54501cd-tls-certs\") pod \"f363c646-16d3-4aff-ac09-9178b54501cd\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") "
Sep 17 18:02:39.721032 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.720819    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-tmp-dir\") pod \"f363c646-16d3-4aff-ac09-9178b54501cd\" (UID: \"f363c646-16d3-4aff-ac09-9178b54501cd\") "
Sep 17 18:02:39.721032 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.721005    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f363c646-16d3-4aff-ac09-9178b54501cd" (UID: "f363c646-16d3-4aff-ac09-9178b54501cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.721186 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.721082    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f363c646-16d3-4aff-ac09-9178b54501cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f363c646-16d3-4aff-ac09-9178b54501cd" (UID: "f363c646-16d3-4aff-ac09-9178b54501cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:39.721275 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.721239    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-home" (OuterVolumeSpecName: "home") pod "f363c646-16d3-4aff-ac09-9178b54501cd" (UID: "f363c646-16d3-4aff-ac09-9178b54501cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.721457 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.721373    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f363c646-16d3-4aff-ac09-9178b54501cd" (UID: "f363c646-16d3-4aff-ac09-9178b54501cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.723090 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.723057    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f363c646-16d3-4aff-ac09-9178b54501cd-kube-api-access-w6hls" (OuterVolumeSpecName: "kube-api-access-w6hls") pod "f363c646-16d3-4aff-ac09-9178b54501cd" (UID: "f363c646-16d3-4aff-ac09-9178b54501cd"). InnerVolumeSpecName "kube-api-access-w6hls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:39.723090 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.723079    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-dshm" (OuterVolumeSpecName: "dshm") pod "f363c646-16d3-4aff-ac09-9178b54501cd" (UID: "f363c646-16d3-4aff-ac09-9178b54501cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.723260 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.723154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f363c646-16d3-4aff-ac09-9178b54501cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f363c646-16d3-4aff-ac09-9178b54501cd" (UID: "f363c646-16d3-4aff-ac09-9178b54501cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:39.742649 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.742620    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f363c646-16d3-4aff-ac09-9178b54501cd" (UID: "f363c646-16d3-4aff-ac09-9178b54501cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.822112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.822014    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f363c646-16d3-4aff-ac09-9178b54501cd-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.822112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.822044    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.822112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.822052    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.822112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.822061    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.822112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.822072    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f363c646-16d3-4aff-ac09-9178b54501cd-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.822112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.822080    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6hls\" (UniqueName: \"kubernetes.io/projected/f363c646-16d3-4aff-ac09-9178b54501cd-kube-api-access-w6hls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.822112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.822089    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.822112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:39.822099    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f363c646-16d3-4aff-ac09-9178b54501cd-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:02:40.318474 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:40.318434    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"
Sep 17 18:02:40.318947 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:40.318429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw" event={"ID":"f363c646-16d3-4aff-ac09-9178b54501cd","Type":"ContainerDied","Data":"8a03fc2e4d9907f7dd09fb35881b62615bae6b3113267ff13e7d6d59ae762fc3"}
Sep 17 18:02:40.318947 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:40.318619    2565 scope.go:117] "RemoveContainer" containerID="63e346b39eeef2fadeee4c06913f3bcbc6624072b162e84a9ea59a736ddac724"
Sep 17 18:02:40.328101 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:40.328083    2565 scope.go:117] "RemoveContainer" containerID="ab60d694648e9aee0613aea33083ef95cb4402d8e0df0fd38187288c345dc870"
Sep 17 18:02:40.341721 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:40.341693    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"]
Sep 17 18:02:40.344588 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:40.344568    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-ctsmw"]
Sep 17 18:02:41.349917 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:41.349884    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" path="/var/lib/kubelet/pods/f363c646-16d3-4aff-ac09-9178b54501cd/volumes"
Sep 17 18:02:43.423804 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:43.423762    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:02:45.965998 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:45.965950    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:53.424110 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:53.424064    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:02:53.897645 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:53.897553    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerName="main" containerID="cri-o://627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899" gracePeriod=45
Sep 17 18:02:55.966464 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:02:55.966415    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:03:03.423877 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:03.423838    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:03:05.965648 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:05.965613    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:03:05.966047 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:05.965716    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:03:13.424107 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:13.424008    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:03:15.966240 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:15.966201    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:03:23.424282 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:23.424232    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:03:24.143219 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.143190    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:03:24.184966 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.184936    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r7d8\" (UniqueName: \"kubernetes.io/projected/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-kube-api-access-6r7d8\") pod \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\" (UID: \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\") "
Sep 17 18:03:24.185079 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.185037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-tls-certs\") pod \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\" (UID: \"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac\") "
Sep 17 18:03:24.187049 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.187020    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-kube-api-access-6r7d8" (OuterVolumeSpecName: "kube-api-access-6r7d8") pod "e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" (UID: "e3b61d7e-0bcf-45d4-887e-4472ab8f0aac"). InnerVolumeSpecName "kube-api-access-6r7d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:24.187139 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.187088    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" (UID: "e3b61d7e-0bcf-45d4-887e-4472ab8f0aac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:24.286011 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.285980    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:03:24.286011 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.286008    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r7d8\" (UniqueName: \"kubernetes.io/projected/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac-kube-api-access-6r7d8\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:03:24.495126 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.495040    2565 generic.go:356] "Generic (PLEG): container finished" podID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerID="627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899" exitCode=0
Sep 17 18:03:24.495126 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.495083    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"
Sep 17 18:03:24.495611 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.495121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" event={"ID":"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac","Type":"ContainerDied","Data":"627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899"}
Sep 17 18:03:24.495611 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.495178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l" event={"ID":"e3b61d7e-0bcf-45d4-887e-4472ab8f0aac","Type":"ContainerDied","Data":"a9ee896e6232af42a6a240d686f7040e4547721f42428ac944511c4476821b09"}
Sep 17 18:03:24.495611 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.495203    2565 scope.go:117] "RemoveContainer" containerID="627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899"
Sep 17 18:03:24.503901 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.503884    2565 scope.go:117] "RemoveContainer" containerID="627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899"
Sep 17 18:03:24.504213 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:03:24.504193    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899\": container with ID starting with 627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899 not found: ID does not exist" containerID="627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899"
Sep 17 18:03:24.504269 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.504222    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899"} err="failed to get container status \"627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899\": rpc error: code = NotFound desc = could not find container \"627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899\": container with ID starting with 627514f10d996e73c2a74fc035b3eafced5d2012a356c382f2d653fe99cf7899 not found: ID does not exist"
Sep 17 18:03:24.528601 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.528566    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"]
Sep 17 18:03:24.533635 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:24.533610    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69q8g7l"]
Sep 17 18:03:25.348737 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:25.348698    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" path="/var/lib/kubelet/pods/e3b61d7e-0bcf-45d4-887e-4472ab8f0aac/volumes"
Sep 17 18:03:33.423860 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:33.423825    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:03:42.417302 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417268    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"]
Sep 17 18:03:42.417704 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417651    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" containerName="storage-initializer"
Sep 17 18:03:42.417704 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417673    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" containerName="storage-initializer"
Sep 17 18:03:42.417784 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417707    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" containerName="main"
Sep 17 18:03:42.417784 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417717    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" containerName="main"
Sep 17 18:03:42.417784 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417736    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerName="main"
Sep 17 18:03:42.417784 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417744    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerName="main"
Sep 17 18:03:42.417922 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417880    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b61d7e-0bcf-45d4-887e-4472ab8f0aac" containerName="main"
Sep 17 18:03:42.417922 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.417893    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f363c646-16d3-4aff-ac09-9178b54501cd" containerName="main"
Sep 17 18:03:42.420682 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.420666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.423631 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.423605    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 18:03:42.423763 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.423645    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-2c8nj\""
Sep 17 18:03:42.423763 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.423654    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:03:42.423763 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.423664    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 18:03:42.423763 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.423670    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 18:03:42.423763 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.423607    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 18:03:42.492558 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.492526    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"]
Sep 17 18:03:42.539136 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.539107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-dshm\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.539293 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.539142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-model-cache\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.539293 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.539164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.539293 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.539257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7mpq\" (UniqueName: \"kubernetes.io/projected/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kube-api-access-f7mpq\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.539477 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.539328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.539477 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.539376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265edc1e-95ec-48db-98a5-2ee902a0ed4e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.539477 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.539414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-home\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.539602 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.539500    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640470 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640653 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265edc1e-95ec-48db-98a5-2ee902a0ed4e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640653 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-home\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640653 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640831 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-dshm\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640831 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-model-cache\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640831 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640982 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7mpq\" (UniqueName: \"kubernetes.io/projected/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kube-api-access-f7mpq\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640982 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.640982 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.640967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.641262 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.641227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-home\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.641262 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.641251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265edc1e-95ec-48db-98a5-2ee902a0ed4e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.641432 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.641271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-model-cache\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.643127 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.643106    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-dshm\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.643532 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.643511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.655701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.655680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7mpq\" (UniqueName: \"kubernetes.io/projected/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kube-api-access-f7mpq\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9qssd\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.686783 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.686715    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"]
Sep 17 18:03:42.689655 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.689623    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:42.692269 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.692251    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-vw6lg\""
Sep 17 18:03:42.705835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.705813    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"]
Sep 17 18:03:42.731122 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.731103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:42.742162 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.742142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvvb\" (UniqueName: \"kubernetes.io/projected/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-kube-api-access-gmvvb\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578592f89\" (UID: \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:42.742257 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.742180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578592f89\" (UID: \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:42.843014 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.842981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvvb\" (UniqueName: \"kubernetes.io/projected/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-kube-api-access-gmvvb\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578592f89\" (UID: \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:42.843014 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.843016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578592f89\" (UID: \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:42.845433 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.845410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578592f89\" (UID: \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:42.851554 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.851537    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvvb\" (UniqueName: \"kubernetes.io/projected/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-kube-api-access-gmvvb\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578592f89\" (UID: \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:42.999921 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:42.999837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:43.013331 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.013303    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"]
Sep 17 18:03:43.016086 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:03:43.016062    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265edc1e_95ec_48db_98a5_2ee902a0ed4e.slice/crio-fc31f06604be135aba8ea0c7240b3d98a68a7830b2932d39ef246cad09fd4f49 WatchSource:0}: Error finding container fc31f06604be135aba8ea0c7240b3d98a68a7830b2932d39ef246cad09fd4f49: Status 404 returned error can't find the container with id fc31f06604be135aba8ea0c7240b3d98a68a7830b2932d39ef246cad09fd4f49
Sep 17 18:03:43.312342 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.312312    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"]
Sep 17 18:03:43.313637 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:03:43.313610    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb001b6f_b3d3_4e15_b1d8_7ae45d58fbb7.slice/crio-b8ca296e4ecb2923eb307bf80f186155f5274adde929b2beab2f45c551fda83e WatchSource:0}: Error finding container b8ca296e4ecb2923eb307bf80f186155f5274adde929b2beab2f45c551fda83e: Status 404 returned error can't find the container with id b8ca296e4ecb2923eb307bf80f186155f5274adde929b2beab2f45c551fda83e
Sep 17 18:03:43.434767 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.434736    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:03:43.443705 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.443678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:03:43.583069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.582977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" event={"ID":"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7","Type":"ContainerStarted","Data":"f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b"}
Sep 17 18:03:43.583069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.583034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" event={"ID":"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7","Type":"ContainerStarted","Data":"b8ca296e4ecb2923eb307bf80f186155f5274adde929b2beab2f45c551fda83e"}
Sep 17 18:03:43.583069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.583062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:03:43.584839 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.584813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" event={"ID":"265edc1e-95ec-48db-98a5-2ee902a0ed4e","Type":"ContainerStarted","Data":"895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009"}
Sep 17 18:03:43.584953 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.584861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" event={"ID":"265edc1e-95ec-48db-98a5-2ee902a0ed4e","Type":"ContainerStarted","Data":"fc31f06604be135aba8ea0c7240b3d98a68a7830b2932d39ef246cad09fd4f49"}
Sep 17 18:03:43.609967 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:43.609917    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" podStartSLOduration=1.6098999200000002 podStartE2EDuration="1.60989992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:43.608808711 +0000 UTC m=+1366.444577934" watchObservedRunningTime="2026-09-17 18:03:43.60989992 +0000 UTC m=+1366.445669142"
Sep 17 18:03:45.594002 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:45.593922    2565 generic.go:356] "Generic (PLEG): container finished" podID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerID="895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009" exitCode=0
Sep 17 18:03:45.594338 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:45.593993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" event={"ID":"265edc1e-95ec-48db-98a5-2ee902a0ed4e","Type":"ContainerDied","Data":"895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009"}
Sep 17 18:03:46.599118 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:46.599083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" event={"ID":"265edc1e-95ec-48db-98a5-2ee902a0ed4e","Type":"ContainerStarted","Data":"df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c"}
Sep 17 18:03:46.625166 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:46.625118    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" podStartSLOduration=2.401656111 podStartE2EDuration="4.625103848s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.223447737s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:46.623544183 +0000 UTC m=+1369.459313430" watchObservedRunningTime="2026-09-17 18:03:46.625103848 +0000 UTC m=+1369.460873071"
Sep 17 18:03:46.731606 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:46.731579    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:52.731646 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:52.731611    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:52.744113 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:52.744086    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:53.635074 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:53.635039    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:03:57.390954 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:03:57.390920    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"]
Sep 17 18:04:12.412279 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:12.412243    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" containerID="cri-o://719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b" gracePeriod=45
Sep 17 18:04:12.435632 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:12.435594    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:04:13.150634 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.150612    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:04:13.191823 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.191761    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-home\") pod \"624c2094-bf22-4925-bb42-51b18fe39697\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") "
Sep 17 18:04:13.191946 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.191826    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624c2094-bf22-4925-bb42-51b18fe39697-tls-certs\") pod \"624c2094-bf22-4925-bb42-51b18fe39697\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") "
Sep 17 18:04:13.191946 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.191850    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-tmp-dir\") pod \"624c2094-bf22-4925-bb42-51b18fe39697\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") "
Sep 17 18:04:13.191946 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.191910    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-dshm\") pod \"624c2094-bf22-4925-bb42-51b18fe39697\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") "
Sep 17 18:04:13.191946 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.191938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrs48\" (UniqueName: \"kubernetes.io/projected/624c2094-bf22-4925-bb42-51b18fe39697-kube-api-access-hrs48\") pod \"624c2094-bf22-4925-bb42-51b18fe39697\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") "
Sep 17 18:04:13.192151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.191971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-kserve-provision-location\") pod \"624c2094-bf22-4925-bb42-51b18fe39697\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") "
Sep 17 18:04:13.192151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.191999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-model-cache\") pod \"624c2094-bf22-4925-bb42-51b18fe39697\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") "
Sep 17 18:04:13.192151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.192055    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624c2094-bf22-4925-bb42-51b18fe39697-cabundle-cert\") pod \"624c2094-bf22-4925-bb42-51b18fe39697\" (UID: \"624c2094-bf22-4925-bb42-51b18fe39697\") "
Sep 17 18:04:13.192755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.192702    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-model-cache" (OuterVolumeSpecName: "model-cache") pod "624c2094-bf22-4925-bb42-51b18fe39697" (UID: "624c2094-bf22-4925-bb42-51b18fe39697"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:13.192755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.192713    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624c2094-bf22-4925-bb42-51b18fe39697-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "624c2094-bf22-4925-bb42-51b18fe39697" (UID: "624c2094-bf22-4925-bb42-51b18fe39697"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:13.192930 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.192451    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-home" (OuterVolumeSpecName: "home") pod "624c2094-bf22-4925-bb42-51b18fe39697" (UID: "624c2094-bf22-4925-bb42-51b18fe39697"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:13.194441 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.194383    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624c2094-bf22-4925-bb42-51b18fe39697-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "624c2094-bf22-4925-bb42-51b18fe39697" (UID: "624c2094-bf22-4925-bb42-51b18fe39697"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:13.194847 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.194823    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624c2094-bf22-4925-bb42-51b18fe39697-kube-api-access-hrs48" (OuterVolumeSpecName: "kube-api-access-hrs48") pod "624c2094-bf22-4925-bb42-51b18fe39697" (UID: "624c2094-bf22-4925-bb42-51b18fe39697"). InnerVolumeSpecName "kube-api-access-hrs48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:13.194934 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.194857    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-dshm" (OuterVolumeSpecName: "dshm") pod "624c2094-bf22-4925-bb42-51b18fe39697" (UID: "624c2094-bf22-4925-bb42-51b18fe39697"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:13.205558 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.205534    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "624c2094-bf22-4925-bb42-51b18fe39697" (UID: "624c2094-bf22-4925-bb42-51b18fe39697"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:13.216044 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.216018    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "624c2094-bf22-4925-bb42-51b18fe39697" (UID: "624c2094-bf22-4925-bb42-51b18fe39697"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:13.293234 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.293201    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:13.293234 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.293226    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624c2094-bf22-4925-bb42-51b18fe39697-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:13.293234 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.293236    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:13.293500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.293243    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:13.293500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.293252    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrs48\" (UniqueName: \"kubernetes.io/projected/624c2094-bf22-4925-bb42-51b18fe39697-kube-api-access-hrs48\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:13.293500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.293262    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:13.293500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.293270    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624c2094-bf22-4925-bb42-51b18fe39697-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:13.293500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.293280    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624c2094-bf22-4925-bb42-51b18fe39697-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:13.699923 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.699881    2565 generic.go:356] "Generic (PLEG): container finished" podID="624c2094-bf22-4925-bb42-51b18fe39697" containerID="719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b" exitCode=0
Sep 17 18:04:13.700349 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.699941    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"
Sep 17 18:04:13.700349 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.699960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" event={"ID":"624c2094-bf22-4925-bb42-51b18fe39697","Type":"ContainerDied","Data":"719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b"}
Sep 17 18:04:13.700349 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.700014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x" event={"ID":"624c2094-bf22-4925-bb42-51b18fe39697","Type":"ContainerDied","Data":"84e235c8f0f52d2f6b971c003de9fa656d92ddbe414fc5c9664037b72a849aec"}
Sep 17 18:04:13.700349 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.700034    2565 scope.go:117] "RemoveContainer" containerID="719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b"
Sep 17 18:04:13.708167 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.708147    2565 scope.go:117] "RemoveContainer" containerID="c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3"
Sep 17 18:04:13.715693 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.715676    2565 scope.go:117] "RemoveContainer" containerID="719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b"
Sep 17 18:04:13.715993 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:04:13.715970    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b\": container with ID starting with 719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b not found: ID does not exist" containerID="719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b"
Sep 17 18:04:13.716091 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.716001    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b"} err="failed to get container status \"719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b\": rpc error: code = NotFound desc = could not find container \"719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b\": container with ID starting with 719decc9e2d3796a18438c50cee31e294141e3864061fe9b55d1d8486985e51b not found: ID does not exist"
Sep 17 18:04:13.716091 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.716026    2565 scope.go:117] "RemoveContainer" containerID="c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3"
Sep 17 18:04:13.716316 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:04:13.716300    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3\": container with ID starting with c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3 not found: ID does not exist" containerID="c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3"
Sep 17 18:04:13.716369 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.716322    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3"} err="failed to get container status \"c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3\": rpc error: code = NotFound desc = could not find container \"c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3\": container with ID starting with c7f409716222b66fc83e9bee42467daf82e29b384c1b2d1c8bac3ad480ad87f3 not found: ID does not exist"
Sep 17 18:04:13.717871 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.717852    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"]
Sep 17 18:04:13.721946 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:13.721927    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-67978d878f-hqk9x"]
Sep 17 18:04:14.590306 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:14.590280    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:04:15.348405 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:15.348372    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="624c2094-bf22-4925-bb42-51b18fe39697" path="/var/lib/kubelet/pods/624c2094-bf22-4925-bb42-51b18fe39697/volumes"
Sep 17 18:04:20.799999 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:20.799965    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"]
Sep 17 18:04:20.814770 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:20.814741    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"]
Sep 17 18:04:20.824230 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:04:20.824192    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:04:20.824230 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:20.824230 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:20.824230 ip-10-0-129-186 kubenswrapper[2565]:  > containerID="df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c" cmd=["/bin/sleep","15"]
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:20.824244    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" message="" err=<
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]:  >
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:04:20.824256    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" podUID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerName="main" containerID="cri-o://df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c"
Sep 17 18:04:20.824546 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:20.824292    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" podUID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerName="main" containerID="cri-o://df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c" gracePeriod=60
Sep 17 18:04:21.171858 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.171836    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:04:21.256881 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.256857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-model-cache\") pod \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") "
Sep 17 18:04:21.256881 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.256885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-home\") pod \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") "
Sep 17 18:04:21.257069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.256914    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tmp-dir\") pod \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") "
Sep 17 18:04:21.257069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.256950    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tls-certs\") pod \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") "
Sep 17 18:04:21.257069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.256978    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-dshm\") pod \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") "
Sep 17 18:04:21.257069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257003    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7mpq\" (UniqueName: \"kubernetes.io/projected/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kube-api-access-f7mpq\") pod \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") "
Sep 17 18:04:21.257069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257041    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kserve-provision-location\") pod \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") "
Sep 17 18:04:21.257324 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257082    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265edc1e-95ec-48db-98a5-2ee902a0ed4e-cabundle-cert\") pod \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\" (UID: \"265edc1e-95ec-48db-98a5-2ee902a0ed4e\") "
Sep 17 18:04:21.257324 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257101    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "265edc1e-95ec-48db-98a5-2ee902a0ed4e" (UID: "265edc1e-95ec-48db-98a5-2ee902a0ed4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.257324 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257126    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-home" (OuterVolumeSpecName: "home") pod "265edc1e-95ec-48db-98a5-2ee902a0ed4e" (UID: "265edc1e-95ec-48db-98a5-2ee902a0ed4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.257324 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "265edc1e-95ec-48db-98a5-2ee902a0ed4e" (UID: "265edc1e-95ec-48db-98a5-2ee902a0ed4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.257526 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257352    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.257526 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257365    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.257526 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257374    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.257526 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.257461    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265edc1e-95ec-48db-98a5-2ee902a0ed4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "265edc1e-95ec-48db-98a5-2ee902a0ed4e" (UID: "265edc1e-95ec-48db-98a5-2ee902a0ed4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:21.259243 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.259218    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-dshm" (OuterVolumeSpecName: "dshm") pod "265edc1e-95ec-48db-98a5-2ee902a0ed4e" (UID: "265edc1e-95ec-48db-98a5-2ee902a0ed4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.259549 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.259521    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "265edc1e-95ec-48db-98a5-2ee902a0ed4e" (UID: "265edc1e-95ec-48db-98a5-2ee902a0ed4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:21.259626 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.259596    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kube-api-access-f7mpq" (OuterVolumeSpecName: "kube-api-access-f7mpq") pod "265edc1e-95ec-48db-98a5-2ee902a0ed4e" (UID: "265edc1e-95ec-48db-98a5-2ee902a0ed4e"). InnerVolumeSpecName "kube-api-access-f7mpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:21.277207 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.277184    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "265edc1e-95ec-48db-98a5-2ee902a0ed4e" (UID: "265edc1e-95ec-48db-98a5-2ee902a0ed4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.358160 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.358142    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265edc1e-95ec-48db-98a5-2ee902a0ed4e-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.358160 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.358161    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.358268 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.358169    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7mpq\" (UniqueName: \"kubernetes.io/projected/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kube-api-access-f7mpq\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.358268 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.358177    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265edc1e-95ec-48db-98a5-2ee902a0ed4e-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.358268 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.358186    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265edc1e-95ec-48db-98a5-2ee902a0ed4e-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.734009 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.733972    2565 generic.go:356] "Generic (PLEG): container finished" podID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerID="df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c" exitCode=0
Sep 17 18:04:21.734204 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.734030    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"
Sep 17 18:04:21.734204 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.734050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" event={"ID":"265edc1e-95ec-48db-98a5-2ee902a0ed4e","Type":"ContainerDied","Data":"df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c"}
Sep 17 18:04:21.734204 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.734103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd" event={"ID":"265edc1e-95ec-48db-98a5-2ee902a0ed4e","Type":"ContainerDied","Data":"fc31f06604be135aba8ea0c7240b3d98a68a7830b2932d39ef246cad09fd4f49"}
Sep 17 18:04:21.734204 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.734126    2565 scope.go:117] "RemoveContainer" containerID="df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c"
Sep 17 18:04:21.742572 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.742552    2565 scope.go:117] "RemoveContainer" containerID="895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009"
Sep 17 18:04:21.749554 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.749534    2565 scope.go:117] "RemoveContainer" containerID="df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c"
Sep 17 18:04:21.749825 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:04:21.749806    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c\": container with ID starting with df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c not found: ID does not exist" containerID="df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c"
Sep 17 18:04:21.749870 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.749839    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c"} err="failed to get container status \"df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c\": rpc error: code = NotFound desc = could not find container \"df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c\": container with ID starting with df73d245b83250b3c16d0546a5b0e4c432985b6af2ce0c527ee8fa1a4688451c not found: ID does not exist"
Sep 17 18:04:21.749870 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.749860    2565 scope.go:117] "RemoveContainer" containerID="895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009"
Sep 17 18:04:21.750080 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:04:21.750063    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009\": container with ID starting with 895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009 not found: ID does not exist" containerID="895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009"
Sep 17 18:04:21.750119 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.750084    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009"} err="failed to get container status \"895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009\": rpc error: code = NotFound desc = could not find container \"895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009\": container with ID starting with 895e702de422a0819ec9d5ea629970b1593107966dd4c36931ee22f220518009 not found: ID does not exist"
Sep 17 18:04:21.755349 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.755328    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"]
Sep 17 18:04:21.762790 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:21.762770    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9qssd"]
Sep 17 18:04:23.347949 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:23.347914    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" path="/var/lib/kubelet/pods/265edc1e-95ec-48db-98a5-2ee902a0ed4e/volumes"
Sep 17 18:04:24.590750 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:24.590714    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:34.590546 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:34.590502    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:35.818822 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:35.818785    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main" containerID="cri-o://f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b" gracePeriod=45
Sep 17 18:04:44.590229 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:44.590131    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:44.590733 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:44.590272    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:04:53.265474 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265439    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"]
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265704    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="storage-initializer"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265718    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="storage-initializer"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265730    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265736    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265748    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerName="storage-initializer"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265753    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerName="storage-initializer"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265767    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerName="main"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265772    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerName="main"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265817    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="624c2094-bf22-4925-bb42-51b18fe39697" containerName="main"
Sep 17 18:04:53.265825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.265827    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="265edc1e-95ec-48db-98a5-2ee902a0ed4e" containerName="main"
Sep 17 18:04:53.268002 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.267986    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:53.271651 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.271622    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-xsnlj\""
Sep 17 18:04:53.271805 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.271668    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:04:53.271805 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.271706    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:04:53.271805 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.271711    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:04:53.271805 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.271800    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-h862r\""
Sep 17 18:04:53.283590 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.283570    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"]
Sep 17 18:04:53.397598 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.397566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkbm\" (UniqueName: \"kubernetes.io/projected/772d65d4-d387-4a08-98f9-b09fc9d44119-kube-api-access-dxkbm\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb\" (UID: \"772d65d4-d387-4a08-98f9-b09fc9d44119\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:53.397765 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.397649    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d65d4-d387-4a08-98f9-b09fc9d44119-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb\" (UID: \"772d65d4-d387-4a08-98f9-b09fc9d44119\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:53.498482 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.498451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkbm\" (UniqueName: \"kubernetes.io/projected/772d65d4-d387-4a08-98f9-b09fc9d44119-kube-api-access-dxkbm\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb\" (UID: \"772d65d4-d387-4a08-98f9-b09fc9d44119\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:53.498655 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.498559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d65d4-d387-4a08-98f9-b09fc9d44119-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb\" (UID: \"772d65d4-d387-4a08-98f9-b09fc9d44119\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:53.501017 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.500996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d65d4-d387-4a08-98f9-b09fc9d44119-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb\" (UID: \"772d65d4-d387-4a08-98f9-b09fc9d44119\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:53.507146 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.507128    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkbm\" (UniqueName: \"kubernetes.io/projected/772d65d4-d387-4a08-98f9-b09fc9d44119-kube-api-access-dxkbm\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb\" (UID: \"772d65d4-d387-4a08-98f9-b09fc9d44119\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:53.578883 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.578805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:53.859634 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:53.859546    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"]
Sep 17 18:04:54.590704 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:54.590668    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:54.858830 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:54.858744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" event={"ID":"772d65d4-d387-4a08-98f9-b09fc9d44119","Type":"ContainerStarted","Data":"e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f"}
Sep 17 18:04:54.858830 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:54.858799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" event={"ID":"772d65d4-d387-4a08-98f9-b09fc9d44119","Type":"ContainerStarted","Data":"ce32270c29ea27aeab08721b9f99e918e0b51bc265f9423df8f5b71fdd90120a"}
Sep 17 18:04:54.858830 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:54.858827    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:04:54.900436 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:04:54.900377    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" podStartSLOduration=1.900361154 podStartE2EDuration="1.900361154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:54.898267931 +0000 UTC m=+1437.734037154" watchObservedRunningTime="2026-09-17 18:04:54.900361154 +0000 UTC m=+1437.736130384"
Sep 17 18:05:04.590339 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:04.590297    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:05:06.061362 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.061338    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:05:06.190719 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.190688    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-tls-certs\") pod \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\" (UID: \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\") "
Sep 17 18:05:06.190907 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.190796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmvvb\" (UniqueName: \"kubernetes.io/projected/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-kube-api-access-gmvvb\") pod \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\" (UID: \"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7\") "
Sep 17 18:05:06.192835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.192803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" (UID: "cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:06.192934 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.192848    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-kube-api-access-gmvvb" (OuterVolumeSpecName: "kube-api-access-gmvvb") pod "cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" (UID: "cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7"). InnerVolumeSpecName "kube-api-access-gmvvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:06.291695 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.291663    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.291695 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.291687    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvvb\" (UniqueName: \"kubernetes.io/projected/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7-kube-api-access-gmvvb\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.902190 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.902159    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerID="f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b" exitCode=0
Sep 17 18:05:06.902370 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.902202    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"
Sep 17 18:05:06.902370 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.902203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" event={"ID":"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7","Type":"ContainerDied","Data":"f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b"}
Sep 17 18:05:06.902370 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.902338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89" event={"ID":"cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7","Type":"ContainerDied","Data":"b8ca296e4ecb2923eb307bf80f186155f5274adde929b2beab2f45c551fda83e"}
Sep 17 18:05:06.902370 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.902366    2565 scope.go:117] "RemoveContainer" containerID="f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b"
Sep 17 18:05:06.911072 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.911053    2565 scope.go:117] "RemoveContainer" containerID="f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b"
Sep 17 18:05:06.911326 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:05:06.911309    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b\": container with ID starting with f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b not found: ID does not exist" containerID="f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b"
Sep 17 18:05:06.911381 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.911336    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b"} err="failed to get container status \"f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b\": rpc error: code = NotFound desc = could not find container \"f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b\": container with ID starting with f23d38d5752a834823a2da4e178497e6fc4ab409f2bf9e5c360b69df8329f38b not found: ID does not exist"
Sep 17 18:05:06.923282 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.923260    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"]
Sep 17 18:05:06.928951 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:06.928928    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578592f89"]
Sep 17 18:05:07.348040 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:07.348005    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" path="/var/lib/kubelet/pods/cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7/volumes"
Sep 17 18:05:25.864478 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:25.864448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:05:39.046942 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.046909    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"]
Sep 17 18:05:39.047421 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.047263    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main"
Sep 17 18:05:39.047421 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.047278    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main"
Sep 17 18:05:39.047421 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.047329    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb001b6f-b3d3-4e15-b1d8-7ae45d58fbb7" containerName="main"
Sep 17 18:05:39.049459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.049437    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.051825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.051803    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-46sls\""
Sep 17 18:05:39.052428 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.052390    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:05:39.052524 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.052430    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:05:39.052524 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.052475    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:05:39.052524 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.052436    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:05:39.052757 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.052390    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:05:39.068255 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.068231    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"]
Sep 17 18:05:39.135691 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.135656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-dshm\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.135886 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.135702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae2fcb0a-4fba-4818-a95b-d18f260e537b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.135886 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.135728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.135886 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.135784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.135886 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.135816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrzd\" (UniqueName: \"kubernetes.io/projected/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kube-api-access-gqrzd\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.135886 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.135839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-home\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.135886 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.135859    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tls-certs\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.136127 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.135890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-model-cache\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.236744 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.236714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-dshm\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.236936 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.236756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae2fcb0a-4fba-4818-a95b-d18f260e537b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.236936 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.236783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.236936 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.236822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.236936 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.236849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrzd\" (UniqueName: \"kubernetes.io/projected/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kube-api-access-gqrzd\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.236936 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.236872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-home\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.237192 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.236969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tls-certs\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.237192 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.237026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-model-cache\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.237304 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.237242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.237304 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.237275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.237438 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.237328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-home\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.237581 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.237555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae2fcb0a-4fba-4818-a95b-d18f260e537b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.237709 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.237689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-model-cache\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.239127 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.239105    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-dshm\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.239406 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.239375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tls-certs\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.245370 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.245345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrzd\" (UniqueName: \"kubernetes.io/projected/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kube-api-access-gqrzd\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.359981 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.359894    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:39.635913 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:39.635822    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"]
Sep 17 18:05:39.638629 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:05:39.638600    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2fcb0a_4fba_4818_a95b_d18f260e537b.slice/crio-cb8c0f7caae3823b605aeec0150fbd007bc6dfe34f6b100d366560f5357f3e47 WatchSource:0}: Error finding container cb8c0f7caae3823b605aeec0150fbd007bc6dfe34f6b100d366560f5357f3e47: Status 404 returned error can't find the container with id cb8c0f7caae3823b605aeec0150fbd007bc6dfe34f6b100d366560f5357f3e47
Sep 17 18:05:40.023958 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:40.023920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" event={"ID":"ae2fcb0a-4fba-4818-a95b-d18f260e537b","Type":"ContainerStarted","Data":"cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62"}
Sep 17 18:05:40.024126 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:40.023979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" event={"ID":"ae2fcb0a-4fba-4818-a95b-d18f260e537b","Type":"ContainerStarted","Data":"cb8c0f7caae3823b605aeec0150fbd007bc6dfe34f6b100d366560f5357f3e47"}
Sep 17 18:05:42.032720 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:42.032688    2565 generic.go:356] "Generic (PLEG): container finished" podID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerID="cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62" exitCode=0
Sep 17 18:05:42.032720 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:42.032724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" event={"ID":"ae2fcb0a-4fba-4818-a95b-d18f260e537b","Type":"ContainerDied","Data":"cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62"}
Sep 17 18:05:43.038415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:43.038358    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" event={"ID":"ae2fcb0a-4fba-4818-a95b-d18f260e537b","Type":"ContainerStarted","Data":"ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12"}
Sep 17 18:05:43.081986 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:43.081933    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" podStartSLOduration=1.946224459 podStartE2EDuration="4.081918852s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.135694393s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:43.081092705 +0000 UTC m=+1485.916861930" watchObservedRunningTime="2026-09-17 18:05:43.081918852 +0000 UTC m=+1485.917688074"
Sep 17 18:05:43.360925 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:43.360836    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:49.360316 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:49.360287    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:49.372734 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:49.372705    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:50.079760 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:50.079732    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:05:59.351931 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:59.351901    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:05:59.353074 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:05:59.353055    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:06:17.450151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.450068    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"]
Sep 17 18:06:17.458816 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:06:17.458760    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:06:17.458816 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:17.458816 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:17.458816 ip-10-0-129-186 kubenswrapper[2565]:  > containerID="ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12" cmd=["/bin/sleep","15"]
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.458838    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" message="" err=<
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]:  >
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:06:17.458855    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" podUID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerName="main" containerID="cri-o://ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12"
Sep 17 18:06:17.459100 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.458912    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" podUID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerName="main" containerID="cri-o://ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12" gracePeriod=60
Sep 17 18:06:17.794479 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.794457    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:06:17.844705 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844678    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-dshm\") pod \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") "
Sep 17 18:06:17.844844 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-home\") pod \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") "
Sep 17 18:06:17.844844 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844735    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae2fcb0a-4fba-4818-a95b-d18f260e537b-cabundle-cert\") pod \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") "
Sep 17 18:06:17.844844 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-model-cache\") pod \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") "
Sep 17 18:06:17.844844 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844789    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tls-certs\") pod \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") "
Sep 17 18:06:17.844844 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844824    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tmp-dir\") pod \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") "
Sep 17 18:06:17.845101 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844894    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqrzd\" (UniqueName: \"kubernetes.io/projected/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kube-api-access-gqrzd\") pod \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") "
Sep 17 18:06:17.845101 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kserve-provision-location\") pod \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\" (UID: \"ae2fcb0a-4fba-4818-a95b-d18f260e537b\") "
Sep 17 18:06:17.845101 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.845053    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae2fcb0a-4fba-4818-a95b-d18f260e537b" (UID: "ae2fcb0a-4fba-4818-a95b-d18f260e537b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:17.845259 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.845131    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2fcb0a-4fba-4818-a95b-d18f260e537b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae2fcb0a-4fba-4818-a95b-d18f260e537b" (UID: "ae2fcb0a-4fba-4818-a95b-d18f260e537b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:17.845259 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.845181    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae2fcb0a-4fba-4818-a95b-d18f260e537b" (UID: "ae2fcb0a-4fba-4818-a95b-d18f260e537b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:17.845340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.844974    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-home" (OuterVolumeSpecName: "home") pod "ae2fcb0a-4fba-4818-a95b-d18f260e537b" (UID: "ae2fcb0a-4fba-4818-a95b-d18f260e537b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:17.845340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.845302    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae2fcb0a-4fba-4818-a95b-d18f260e537b-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:06:17.845340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.845322    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:06:17.845340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.845336    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:06:17.846938 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.846916    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-dshm" (OuterVolumeSpecName: "dshm") pod "ae2fcb0a-4fba-4818-a95b-d18f260e537b" (UID: "ae2fcb0a-4fba-4818-a95b-d18f260e537b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:17.847038 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.846990    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae2fcb0a-4fba-4818-a95b-d18f260e537b" (UID: "ae2fcb0a-4fba-4818-a95b-d18f260e537b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:17.847283 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.847264    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kube-api-access-gqrzd" (OuterVolumeSpecName: "kube-api-access-gqrzd") pod "ae2fcb0a-4fba-4818-a95b-d18f260e537b" (UID: "ae2fcb0a-4fba-4818-a95b-d18f260e537b"). InnerVolumeSpecName "kube-api-access-gqrzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:17.868031 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.868004    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae2fcb0a-4fba-4818-a95b-d18f260e537b" (UID: "ae2fcb0a-4fba-4818-a95b-d18f260e537b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:17.945844 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.945823    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqrzd\" (UniqueName: \"kubernetes.io/projected/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kube-api-access-gqrzd\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:06:17.945955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.945847    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:06:17.945955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.945856    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:06:17.945955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.945866    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2fcb0a-4fba-4818-a95b-d18f260e537b-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:06:17.945955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:17.945873    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2fcb0a-4fba-4818-a95b-d18f260e537b-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.172069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.172035    2565 generic.go:356] "Generic (PLEG): container finished" podID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerID="ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12" exitCode=0
Sep 17 18:06:18.172251 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.172086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" event={"ID":"ae2fcb0a-4fba-4818-a95b-d18f260e537b","Type":"ContainerDied","Data":"ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12"}
Sep 17 18:06:18.172251 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.172134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz" event={"ID":"ae2fcb0a-4fba-4818-a95b-d18f260e537b","Type":"ContainerDied","Data":"cb8c0f7caae3823b605aeec0150fbd007bc6dfe34f6b100d366560f5357f3e47"}
Sep 17 18:06:18.172251 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.172089    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"
Sep 17 18:06:18.172251 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.172167    2565 scope.go:117] "RemoveContainer" containerID="ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12"
Sep 17 18:06:18.181135 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.181119    2565 scope.go:117] "RemoveContainer" containerID="cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62"
Sep 17 18:06:18.188366 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.188349    2565 scope.go:117] "RemoveContainer" containerID="ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12"
Sep 17 18:06:18.188635 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:06:18.188617    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12\": container with ID starting with ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12 not found: ID does not exist" containerID="ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12"
Sep 17 18:06:18.188687 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.188645    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12"} err="failed to get container status \"ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12\": rpc error: code = NotFound desc = could not find container \"ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12\": container with ID starting with ccef683855bc104b2ce90dfada6bcc61fb4e9ad91cf08110ba39eaeaadb0ee12 not found: ID does not exist"
Sep 17 18:06:18.188687 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.188661    2565 scope.go:117] "RemoveContainer" containerID="cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62"
Sep 17 18:06:18.188898 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:06:18.188880    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62\": container with ID starting with cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62 not found: ID does not exist" containerID="cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62"
Sep 17 18:06:18.188935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.188905    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62"} err="failed to get container status \"cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62\": rpc error: code = NotFound desc = could not find container \"cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62\": container with ID starting with cd64ac1a0dbfe7a03c14c7c6565160bc17d3bbeac65198d22e46848384294e62 not found: ID does not exist"
Sep 17 18:06:18.193177 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.193153    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"]
Sep 17 18:06:18.198334 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:18.198314    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-85mlz"]
Sep 17 18:06:19.348136 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:06:19.348107    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" path="/var/lib/kubelet/pods/ae2fcb0a-4fba-4818-a95b-d18f260e537b/volumes"
Sep 17 18:07:43.976950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:07:43.976870    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"]
Sep 17 18:07:45.863933 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:07:45.863898    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:55.863610 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:07:55.863564    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:58.998662 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:07:58.998622    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main" containerID="cri-o://e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f" gracePeriod=45
Sep 17 18:08:05.863577 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:05.863536    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:05.864023 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:05.863644    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:08:15.864379 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:15.864341    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:25.863944 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:25.863908    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:29.240406 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.240375    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:08:29.343603 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.343515    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxkbm\" (UniqueName: \"kubernetes.io/projected/772d65d4-d387-4a08-98f9-b09fc9d44119-kube-api-access-dxkbm\") pod \"772d65d4-d387-4a08-98f9-b09fc9d44119\" (UID: \"772d65d4-d387-4a08-98f9-b09fc9d44119\") "
Sep 17 18:08:29.343603 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.343557    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d65d4-d387-4a08-98f9-b09fc9d44119-tls-certs\") pod \"772d65d4-d387-4a08-98f9-b09fc9d44119\" (UID: \"772d65d4-d387-4a08-98f9-b09fc9d44119\") "
Sep 17 18:08:29.345936 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.345901    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772d65d4-d387-4a08-98f9-b09fc9d44119-kube-api-access-dxkbm" (OuterVolumeSpecName: "kube-api-access-dxkbm") pod "772d65d4-d387-4a08-98f9-b09fc9d44119" (UID: "772d65d4-d387-4a08-98f9-b09fc9d44119"). InnerVolumeSpecName "kube-api-access-dxkbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:29.346059 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.345984    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772d65d4-d387-4a08-98f9-b09fc9d44119-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "772d65d4-d387-4a08-98f9-b09fc9d44119" (UID: "772d65d4-d387-4a08-98f9-b09fc9d44119"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:29.444887 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.444856    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d65d4-d387-4a08-98f9-b09fc9d44119-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:08:29.444887 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.444886    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxkbm\" (UniqueName: \"kubernetes.io/projected/772d65d4-d387-4a08-98f9-b09fc9d44119-kube-api-access-dxkbm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:08:29.647150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.647064    2565 generic.go:356] "Generic (PLEG): container finished" podID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerID="e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f" exitCode=0
Sep 17 18:08:29.647150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.647104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" event={"ID":"772d65d4-d387-4a08-98f9-b09fc9d44119","Type":"ContainerDied","Data":"e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f"}
Sep 17 18:08:29.647150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.647120    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"
Sep 17 18:08:29.647150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.647152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb" event={"ID":"772d65d4-d387-4a08-98f9-b09fc9d44119","Type":"ContainerDied","Data":"ce32270c29ea27aeab08721b9f99e918e0b51bc265f9423df8f5b71fdd90120a"}
Sep 17 18:08:29.647462 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.647176    2565 scope.go:117] "RemoveContainer" containerID="e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f"
Sep 17 18:08:29.655692 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.655675    2565 scope.go:117] "RemoveContainer" containerID="e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f"
Sep 17 18:08:29.655969 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:08:29.655951    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f\": container with ID starting with e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f not found: ID does not exist" containerID="e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f"
Sep 17 18:08:29.656033 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.655977    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f"} err="failed to get container status \"e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f\": rpc error: code = NotFound desc = could not find container \"e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f\": container with ID starting with e56c8b94bffbb8ce09695e7716cbe9222a76d5218ce179808ff7d4dc200ef90f not found: ID does not exist"
Sep 17 18:08:29.663437 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.663411    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"]
Sep 17 18:08:29.667592 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:29.667570    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qr2tb"]
Sep 17 18:08:31.348689 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:31.348656    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" path="/var/lib/kubelet/pods/772d65d4-d387-4a08-98f9-b09fc9d44119/volumes"
Sep 17 18:08:34.121490 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121462    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"]
Sep 17 18:08:34.121835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121714    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerName="storage-initializer"
Sep 17 18:08:34.121835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121724    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerName="storage-initializer"
Sep 17 18:08:34.121835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121736    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerName="main"
Sep 17 18:08:34.121835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121741    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerName="main"
Sep 17 18:08:34.121835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121754    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main"
Sep 17 18:08:34.121835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121759    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main"
Sep 17 18:08:34.121835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121806    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2fcb0a-4fba-4818-a95b-d18f260e537b" containerName="main"
Sep 17 18:08:34.121835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.121815    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="772d65d4-d387-4a08-98f9-b09fc9d44119" containerName="main"
Sep 17 18:08:34.124698 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.124681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.131081 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.130944    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:08:34.133272 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.133247    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:08:34.133440 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.133303    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:08:34.133569 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.133555    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-7khjq\""
Sep 17 18:08:34.161848 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.161821    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"]
Sep 17 18:08:34.280990 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.280960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.281158 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.281007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.281158 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.281026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.281158 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.281089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95kc\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-kube-api-access-b95kc\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.281158 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.281128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.281310 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.281163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.281310 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.281185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.281310 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.281212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c768b587-73b7-42e4-9f7a-3280904acc7e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.281310 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.281238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.382563 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.382563 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.382755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.382755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.382755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95kc\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-kube-api-access-b95kc\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.382755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.382755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.382755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.383062 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.382757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c768b587-73b7-42e4-9f7a-3280904acc7e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.383062 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.383018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.383176 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.383055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.383176 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.383129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.383291 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.383187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.383371 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.383349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c768b587-73b7-42e4-9f7a-3280904acc7e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.384867 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.384846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.385220 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.385203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.404333 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.404304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95kc\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-kube-api-access-b95kc\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.411627 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.411598    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-cgm6c\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.434609 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.434581    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:34.711489 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.711457    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"]
Sep 17 18:08:34.715432 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:08:34.715387    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc768b587_73b7_42e4_9f7a_3280904acc7e.slice/crio-ab72c0249536794f1978514641d6fb3ac2cb6f7c8ba27b8eabd901dcba215af7 WatchSource:0}: Error finding container ab72c0249536794f1978514641d6fb3ac2cb6f7c8ba27b8eabd901dcba215af7: Status 404 returned error can't find the container with id ab72c0249536794f1978514641d6fb3ac2cb6f7c8ba27b8eabd901dcba215af7
Sep 17 18:08:34.717282 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.717253    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:08:34.717340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.717331    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:08:34.717378 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:34.717361    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:08:35.670460 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:35.670425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" event={"ID":"c768b587-73b7-42e4-9f7a-3280904acc7e","Type":"ContainerStarted","Data":"f6b17c6449b41179d23b50a90479c7fb0f33b03d4750cc9daf4272ffd2468680"}
Sep 17 18:08:35.670891 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:35.670474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" event={"ID":"c768b587-73b7-42e4-9f7a-3280904acc7e","Type":"ContainerStarted","Data":"ab72c0249536794f1978514641d6fb3ac2cb6f7c8ba27b8eabd901dcba215af7"}
Sep 17 18:08:35.690025 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:35.689971    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" podStartSLOduration=1.689951787 podStartE2EDuration="1.689951787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:35.689047395 +0000 UTC m=+1658.524816630" watchObservedRunningTime="2026-09-17 18:08:35.689951787 +0000 UTC m=+1658.525721024"
Sep 17 18:08:36.435600 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:36.435566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:36.440336 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:36.440313    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:36.674694 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:36.674669    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:36.675563 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:36.675540    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:52.716607 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:52.716555    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:08:53.736202 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:53.736113    2565 generic.go:356] "Generic (PLEG): container finished" podID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerID="f6b17c6449b41179d23b50a90479c7fb0f33b03d4750cc9daf4272ffd2468680" exitCode=0
Sep 17 18:08:53.736754 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:53.736193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" event={"ID":"c768b587-73b7-42e4-9f7a-3280904acc7e","Type":"ContainerDied","Data":"f6b17c6449b41179d23b50a90479c7fb0f33b03d4750cc9daf4272ffd2468680"}
Sep 17 18:08:53.736754 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:53.736673    2565 scope.go:117] "RemoveContainer" containerID="f6b17c6449b41179d23b50a90479c7fb0f33b03d4750cc9daf4272ffd2468680"
Sep 17 18:08:53.737614 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:53.737589    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:08:53.737686 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:53.737656    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:08:53.737722 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:53.737695    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:08:54.741879 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:54.741847    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" event={"ID":"c768b587-73b7-42e4-9f7a-3280904acc7e","Type":"ContainerStarted","Data":"ec7b3ca45091784262be379b456a04973cf1a5b98e11ac4300a3841f319df14a"}
Sep 17 18:08:55.435105 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:55.435071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:08:55.435310 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:55.435287    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 18:08:56.436029 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:56.435989    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 18:08:57.435886 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:57.435844    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 18:08:58.435225 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:58.435187    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 18:08:59.435416 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:08:59.435362    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 18:09:00.438973 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:00.438944    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:09:00.439364 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:00.439182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:09:00.439828 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:00.439812    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:09:05.815390 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.815359    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"]
Sep 17 18:09:05.818217 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.818199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:05.822340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.822318    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:09:05.823608 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.823588    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-zqr5c\""
Sep 17 18:09:05.823697 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.823627    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:09:05.823697 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.823648    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 18:09:05.829704 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.829684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"]
Sep 17 18:09:05.943264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.943225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:05.943264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.943257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:05.943537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.943289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:05.943537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.943314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:05.943537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.943344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965f8420-ace8-4eec-8f10-bfe8a75e301d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:05.943537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:05.943368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xw2v\" (UniqueName: \"kubernetes.io/projected/965f8420-ace8-4eec-8f10-bfe8a75e301d-kube-api-access-5xw2v\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.044535 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.044497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965f8420-ace8-4eec-8f10-bfe8a75e301d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.044711 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.044543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xw2v\" (UniqueName: \"kubernetes.io/projected/965f8420-ace8-4eec-8f10-bfe8a75e301d-kube-api-access-5xw2v\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.044711 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.044565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.044711 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.044584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.044711 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.044622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.044711 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.044639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.045074 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.045051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.045151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.045114    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.045151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.045134    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.046954 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.046933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.047058 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.047046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965f8420-ace8-4eec-8f10-bfe8a75e301d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.056991 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.056966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xw2v\" (UniqueName: \"kubernetes.io/projected/965f8420-ace8-4eec-8f10-bfe8a75e301d-kube-api-access-5xw2v\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.061810 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.061785    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"]
Sep 17 18:09:06.063992 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.063974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.066793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.066743    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-x2k86\""
Sep 17 18:09:06.076287 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.076263    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"]
Sep 17 18:09:06.127955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.127933    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:06.145660 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.145631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dkx\" (UniqueName: \"kubernetes.io/projected/b916ee05-c26f-4234-b50f-d0ff3232c5ac-kube-api-access-x2dkx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx\" (UID: \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.145758 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.145666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916ee05-c26f-4234-b50f-d0ff3232c5ac-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx\" (UID: \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.246718 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.246681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dkx\" (UniqueName: \"kubernetes.io/projected/b916ee05-c26f-4234-b50f-d0ff3232c5ac-kube-api-access-x2dkx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx\" (UID: \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.246718 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.246722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916ee05-c26f-4234-b50f-d0ff3232c5ac-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx\" (UID: \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.249151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.249126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916ee05-c26f-4234-b50f-d0ff3232c5ac-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx\" (UID: \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.256355 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.256336    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dkx\" (UniqueName: \"kubernetes.io/projected/b916ee05-c26f-4234-b50f-d0ff3232c5ac-kube-api-access-x2dkx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx\" (UID: \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.374808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.374730    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.413133 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.413107    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"]
Sep 17 18:09:06.415519 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:09:06.415493    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965f8420_ace8_4eec_8f10_bfe8a75e301d.slice/crio-8e5c23355798eff4eecc6844a7bb65e7d903186d60c79a124af35e9246c982a9 WatchSource:0}: Error finding container 8e5c23355798eff4eecc6844a7bb65e7d903186d60c79a124af35e9246c982a9: Status 404 returned error can't find the container with id 8e5c23355798eff4eecc6844a7bb65e7d903186d60c79a124af35e9246c982a9
Sep 17 18:09:06.650790 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.650713    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"]
Sep 17 18:09:06.654311 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:09:06.654278    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb916ee05_c26f_4234_b50f_d0ff3232c5ac.slice/crio-bf1396afcbb66f72aad021dd07c16835def29e35d3fcef0b325b2b254ba08b7b WatchSource:0}: Error finding container bf1396afcbb66f72aad021dd07c16835def29e35d3fcef0b325b2b254ba08b7b: Status 404 returned error can't find the container with id bf1396afcbb66f72aad021dd07c16835def29e35d3fcef0b325b2b254ba08b7b
Sep 17 18:09:06.794085 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.794048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" event={"ID":"b916ee05-c26f-4234-b50f-d0ff3232c5ac","Type":"ContainerStarted","Data":"bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba"}
Sep 17 18:09:06.794241 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.794105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" event={"ID":"b916ee05-c26f-4234-b50f-d0ff3232c5ac","Type":"ContainerStarted","Data":"bf1396afcbb66f72aad021dd07c16835def29e35d3fcef0b325b2b254ba08b7b"}
Sep 17 18:09:06.794287 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.794273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:09:06.795875 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.795853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" event={"ID":"965f8420-ace8-4eec-8f10-bfe8a75e301d","Type":"ContainerStarted","Data":"e657a167903b191fe24942da9bb25a9b32355ee3a067dc75a68fc8823a749d7c"}
Sep 17 18:09:06.796003 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.795887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" event={"ID":"965f8420-ace8-4eec-8f10-bfe8a75e301d","Type":"ContainerStarted","Data":"8e5c23355798eff4eecc6844a7bb65e7d903186d60c79a124af35e9246c982a9"}
Sep 17 18:09:06.814600 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.814512    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" podStartSLOduration=0.814489049 podStartE2EDuration="814.489049ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:06.813014833 +0000 UTC m=+1689.648784056" watchObservedRunningTime="2026-09-17 18:09:06.814489049 +0000 UTC m=+1689.650258271"
Sep 17 18:09:06.839616 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:06.839561    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" podStartSLOduration=1.8395452209999998 podStartE2EDuration="1.839545221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:06.838036782 +0000 UTC m=+1689.673806025" watchObservedRunningTime="2026-09-17 18:09:06.839545221 +0000 UTC m=+1689.675314487"
Sep 17 18:09:07.128511 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:07.128475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:16.129025 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:16.128936    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:16.141513 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:16.141488    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:16.851509 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:16.851482    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:09:37.801603 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:09:37.801570    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:10:07.092802 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.092766    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"]
Sep 17 18:10:07.095015 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.094998    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:07.099306 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.099281    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 18:10:07.099489 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.099309    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-vngpg\""
Sep 17 18:10:07.099489 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.099308    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-blxwf\""
Sep 17 18:10:07.099489 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.099460    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 18:10:07.099768 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.099750    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 18:10:07.116338 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.116315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"]
Sep 17 18:10:07.160260 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.160226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10270f87-242a-456f-a37b-7e8a6b6a5081-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4\" (UID: \"10270f87-242a-456f-a37b-7e8a6b6a5081\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:07.160423 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.160277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwg6j\" (UniqueName: \"kubernetes.io/projected/10270f87-242a-456f-a37b-7e8a6b6a5081-kube-api-access-xwg6j\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4\" (UID: \"10270f87-242a-456f-a37b-7e8a6b6a5081\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:07.261516 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.261439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10270f87-242a-456f-a37b-7e8a6b6a5081-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4\" (UID: \"10270f87-242a-456f-a37b-7e8a6b6a5081\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:07.261685 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.261569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwg6j\" (UniqueName: \"kubernetes.io/projected/10270f87-242a-456f-a37b-7e8a6b6a5081-kube-api-access-xwg6j\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4\" (UID: \"10270f87-242a-456f-a37b-7e8a6b6a5081\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:07.264196 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.264129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10270f87-242a-456f-a37b-7e8a6b6a5081-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4\" (UID: \"10270f87-242a-456f-a37b-7e8a6b6a5081\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:07.269822 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.269790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwg6j\" (UniqueName: \"kubernetes.io/projected/10270f87-242a-456f-a37b-7e8a6b6a5081-kube-api-access-xwg6j\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4\" (UID: \"10270f87-242a-456f-a37b-7e8a6b6a5081\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:07.406121 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.406037    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:07.681825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:07.681795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"]
Sep 17 18:10:07.684630 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:10:07.684604    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10270f87_242a_456f_a37b_7e8a6b6a5081.slice/crio-f6d6e0ea27fd07b370821ce113017632a4a16f2d53cfc121ea8fa9a16c2a6f9e WatchSource:0}: Error finding container f6d6e0ea27fd07b370821ce113017632a4a16f2d53cfc121ea8fa9a16c2a6f9e: Status 404 returned error can't find the container with id f6d6e0ea27fd07b370821ce113017632a4a16f2d53cfc121ea8fa9a16c2a6f9e
Sep 17 18:10:08.030976 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:08.030893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" event={"ID":"10270f87-242a-456f-a37b-7e8a6b6a5081","Type":"ContainerStarted","Data":"c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2"}
Sep 17 18:10:08.030976 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:08.030951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" event={"ID":"10270f87-242a-456f-a37b-7e8a6b6a5081","Type":"ContainerStarted","Data":"f6d6e0ea27fd07b370821ce113017632a4a16f2d53cfc121ea8fa9a16c2a6f9e"}
Sep 17 18:10:08.031162 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:08.031004    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:08.053524 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:08.053474    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" podStartSLOduration=1.053458378 podStartE2EDuration="1.053458378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:08.052456336 +0000 UTC m=+1750.888225577" watchObservedRunningTime="2026-09-17 18:10:08.053458378 +0000 UTC m=+1750.889227653"
Sep 17 18:10:08.851219 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:08.851187    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"]
Sep 17 18:10:08.851681 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:08.851412    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy" containerID="cri-o://ec7b3ca45091784262be379b456a04973cf1a5b98e11ac4300a3841f319df14a" gracePeriod=30
Sep 17 18:10:14.057604 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.057577    2565 generic.go:356] "Generic (PLEG): container finished" podID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerID="ec7b3ca45091784262be379b456a04973cf1a5b98e11ac4300a3841f319df14a" exitCode=0
Sep 17 18:10:14.057950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.057627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" event={"ID":"c768b587-73b7-42e4-9f7a-3280904acc7e","Type":"ContainerDied","Data":"ec7b3ca45091784262be379b456a04973cf1a5b98e11ac4300a3841f319df14a"}
Sep 17 18:10:14.057950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.057682    2565 scope.go:117] "RemoveContainer" containerID="f6b17c6449b41179d23b50a90479c7fb0f33b03d4750cc9daf4272ffd2468680"
Sep 17 18:10:14.098306 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.098285    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:10:14.220413 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220368    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-token\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220575 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-podinfo\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220575 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-socket\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220575 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220531    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-data\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220745 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220624    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c768b587-73b7-42e4-9f7a-3280904acc7e-istiod-ca-cert\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220745 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220684    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-certs\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220745 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220720    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b95kc\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-kube-api-access-b95kc\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220894 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220728    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:14.220894 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220756    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-credential-socket\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220894 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220782    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-envoy\") pod \"c768b587-73b7-42e4-9f7a-3280904acc7e\" (UID: \"c768b587-73b7-42e4-9f7a-3280904acc7e\") "
Sep 17 18:10:14.220894 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220811    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:14.221107 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.220908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:14.221107 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.221028    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:14.221107 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.221064    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.221107 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.221084    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.221107 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.221098    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.221311 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.221188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c768b587-73b7-42e4-9f7a-3280904acc7e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:14.223129 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.223108    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:14.223129 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.223109    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:10:14.223292 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.223207    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-kube-api-access-b95kc" (OuterVolumeSpecName: "kube-api-access-b95kc") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "kube-api-access-b95kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:14.223292 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.223223    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c768b587-73b7-42e4-9f7a-3280904acc7e" (UID: "c768b587-73b7-42e4-9f7a-3280904acc7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:14.321776 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.321731    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.321776 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.321770    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.321776 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.321779    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c768b587-73b7-42e4-9f7a-3280904acc7e-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.321776 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.321787    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b95kc\" (UniqueName: \"kubernetes.io/projected/c768b587-73b7-42e4-9f7a-3280904acc7e-kube-api-access-b95kc\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.321776 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.321796    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.322110 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:14.321805    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c768b587-73b7-42e4-9f7a-3280904acc7e-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:10:15.062517 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:15.062480    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"
Sep 17 18:10:15.062517 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:15.062488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c" event={"ID":"c768b587-73b7-42e4-9f7a-3280904acc7e","Type":"ContainerDied","Data":"ab72c0249536794f1978514641d6fb3ac2cb6f7c8ba27b8eabd901dcba215af7"}
Sep 17 18:10:15.063042 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:15.062572    2565 scope.go:117] "RemoveContainer" containerID="ec7b3ca45091784262be379b456a04973cf1a5b98e11ac4300a3841f319df14a"
Sep 17 18:10:15.090467 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:15.090438    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"]
Sep 17 18:10:15.096455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:15.096427    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-cgm6c"]
Sep 17 18:10:15.349267 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:15.349191    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" path="/var/lib/kubelet/pods/c768b587-73b7-42e4-9f7a-3280904acc7e/volumes"
Sep 17 18:10:39.037295 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:39.037265    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:10:59.373356 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:59.373326    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:10:59.381517 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:10:59.381486    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:11:12.650522 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:12.650478    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"]
Sep 17 18:11:12.659224 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:11:12.659193    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:11:12.659224 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:12.659224 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:12.659224 ip-10-0-129-186 kubenswrapper[2565]:  > containerID="e657a167903b191fe24942da9bb25a9b32355ee3a067dc75a68fc8823a749d7c" cmd=["/bin/sleep","15"]
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:12.659244    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" message="" err=<
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]:  >
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:11:12.659261    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" podUID="965f8420-ace8-4eec-8f10-bfe8a75e301d" containerName="main" containerID="cri-o://e657a167903b191fe24942da9bb25a9b32355ee3a067dc75a68fc8823a749d7c"
Sep 17 18:11:12.659524 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:12.659300    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" podUID="965f8420-ace8-4eec-8f10-bfe8a75e301d" containerName="main" containerID="cri-o://e657a167903b191fe24942da9bb25a9b32355ee3a067dc75a68fc8823a749d7c" gracePeriod=60
Sep 17 18:11:12.660050 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:12.659970    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"]
Sep 17 18:11:12.841630 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:12.841588    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" podUID="965f8420-ace8-4eec-8f10-bfe8a75e301d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 17 18:11:13.281105 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.281074    2565 generic.go:356] "Generic (PLEG): container finished" podID="965f8420-ace8-4eec-8f10-bfe8a75e301d" containerID="e657a167903b191fe24942da9bb25a9b32355ee3a067dc75a68fc8823a749d7c" exitCode=0
Sep 17 18:11:13.281228 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.281146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" event={"ID":"965f8420-ace8-4eec-8f10-bfe8a75e301d","Type":"ContainerDied","Data":"e657a167903b191fe24942da9bb25a9b32355ee3a067dc75a68fc8823a749d7c"}
Sep 17 18:11:13.401172 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.401150    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:11:13.484354 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484313    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xw2v\" (UniqueName: \"kubernetes.io/projected/965f8420-ace8-4eec-8f10-bfe8a75e301d-kube-api-access-5xw2v\") pod \"965f8420-ace8-4eec-8f10-bfe8a75e301d\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") "
Sep 17 18:11:13.484550 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484362    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965f8420-ace8-4eec-8f10-bfe8a75e301d-tls-certs\") pod \"965f8420-ace8-4eec-8f10-bfe8a75e301d\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") "
Sep 17 18:11:13.484550 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-dshm\") pod \"965f8420-ace8-4eec-8f10-bfe8a75e301d\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") "
Sep 17 18:11:13.484550 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484494    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-tmp-dir\") pod \"965f8420-ace8-4eec-8f10-bfe8a75e301d\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") "
Sep 17 18:11:13.484550 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484540    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-home\") pod \"965f8420-ace8-4eec-8f10-bfe8a75e301d\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") "
Sep 17 18:11:13.484725 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484563    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-model-cache\") pod \"965f8420-ace8-4eec-8f10-bfe8a75e301d\" (UID: \"965f8420-ace8-4eec-8f10-bfe8a75e301d\") "
Sep 17 18:11:13.484848 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484796    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "965f8420-ace8-4eec-8f10-bfe8a75e301d" (UID: "965f8420-ace8-4eec-8f10-bfe8a75e301d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:13.484920 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484868    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-home" (OuterVolumeSpecName: "home") pod "965f8420-ace8-4eec-8f10-bfe8a75e301d" (UID: "965f8420-ace8-4eec-8f10-bfe8a75e301d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:13.484962 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.484918    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-model-cache" (OuterVolumeSpecName: "model-cache") pod "965f8420-ace8-4eec-8f10-bfe8a75e301d" (UID: "965f8420-ace8-4eec-8f10-bfe8a75e301d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:13.486870 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.486841    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-dshm" (OuterVolumeSpecName: "dshm") pod "965f8420-ace8-4eec-8f10-bfe8a75e301d" (UID: "965f8420-ace8-4eec-8f10-bfe8a75e301d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:13.486870 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.486843    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965f8420-ace8-4eec-8f10-bfe8a75e301d-kube-api-access-5xw2v" (OuterVolumeSpecName: "kube-api-access-5xw2v") pod "965f8420-ace8-4eec-8f10-bfe8a75e301d" (UID: "965f8420-ace8-4eec-8f10-bfe8a75e301d"). InnerVolumeSpecName "kube-api-access-5xw2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:13.487014 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.486896    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965f8420-ace8-4eec-8f10-bfe8a75e301d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "965f8420-ace8-4eec-8f10-bfe8a75e301d" (UID: "965f8420-ace8-4eec-8f10-bfe8a75e301d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:13.585537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.585436    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xw2v\" (UniqueName: \"kubernetes.io/projected/965f8420-ace8-4eec-8f10-bfe8a75e301d-kube-api-access-5xw2v\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:11:13.585537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.585505    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965f8420-ace8-4eec-8f10-bfe8a75e301d-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:11:13.585537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.585519    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:11:13.585537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.585529    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:11:13.585537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.585542    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:11:13.585811 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:13.585553    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965f8420-ace8-4eec-8f10-bfe8a75e301d-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:11:14.285586 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:14.285555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m" event={"ID":"965f8420-ace8-4eec-8f10-bfe8a75e301d","Type":"ContainerDied","Data":"8e5c23355798eff4eecc6844a7bb65e7d903186d60c79a124af35e9246c982a9"}
Sep 17 18:11:14.285986 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:14.285609    2565 scope.go:117] "RemoveContainer" containerID="e657a167903b191fe24942da9bb25a9b32355ee3a067dc75a68fc8823a749d7c"
Sep 17 18:11:14.285986 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:14.285564    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"
Sep 17 18:11:14.320661 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:14.320634    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"]
Sep 17 18:11:14.322697 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:14.322676    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rh42m"]
Sep 17 18:11:15.348449 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:15.348416    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="965f8420-ace8-4eec-8f10-bfe8a75e301d" path="/var/lib/kubelet/pods/965f8420-ace8-4eec-8f10-bfe8a75e301d/volumes"
Sep 17 18:11:17.809155 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:17.809118    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:27.678066 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:27.678026    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main" containerID="cri-o://bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba" gracePeriod=45
Sep 17 18:11:27.801740 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:27.801702    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:37.802099 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:37.802060    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:37.802561 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:37.802171    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:11:47.802317 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:47.802278    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:57.800705 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:11:57.800678    2565 logging.go:55] [core] [Channel #650 SubChannel #651]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.41:9003", ServerName: "10.133.0.41:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.41:9003: connect: connection refused"
Sep 17 18:11:57.913048 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:57.913025    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:11:58.017554 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.017464    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2dkx\" (UniqueName: \"kubernetes.io/projected/b916ee05-c26f-4234-b50f-d0ff3232c5ac-kube-api-access-x2dkx\") pod \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\" (UID: \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\") "
Sep 17 18:11:58.017554 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.017520    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916ee05-c26f-4234-b50f-d0ff3232c5ac-tls-certs\") pod \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\" (UID: \"b916ee05-c26f-4234-b50f-d0ff3232c5ac\") "
Sep 17 18:11:58.019657 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.019621    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b916ee05-c26f-4234-b50f-d0ff3232c5ac-kube-api-access-x2dkx" (OuterVolumeSpecName: "kube-api-access-x2dkx") pod "b916ee05-c26f-4234-b50f-d0ff3232c5ac" (UID: "b916ee05-c26f-4234-b50f-d0ff3232c5ac"). InnerVolumeSpecName "kube-api-access-x2dkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:58.019657 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.019644    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b916ee05-c26f-4234-b50f-d0ff3232c5ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b916ee05-c26f-4234-b50f-d0ff3232c5ac" (UID: "b916ee05-c26f-4234-b50f-d0ff3232c5ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:58.118495 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.118455    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916ee05-c26f-4234-b50f-d0ff3232c5ac-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:11:58.118495 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.118486    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2dkx\" (UniqueName: \"kubernetes.io/projected/b916ee05-c26f-4234-b50f-d0ff3232c5ac-kube-api-access-x2dkx\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:11:58.447646 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.447615    2565 generic.go:356] "Generic (PLEG): container finished" podID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerID="bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba" exitCode=0
Sep 17 18:11:58.447808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.447660    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"
Sep 17 18:11:58.447808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.447676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" event={"ID":"b916ee05-c26f-4234-b50f-d0ff3232c5ac","Type":"ContainerDied","Data":"bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba"}
Sep 17 18:11:58.447808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.447714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" event={"ID":"b916ee05-c26f-4234-b50f-d0ff3232c5ac","Type":"ContainerDied","Data":"bf1396afcbb66f72aad021dd07c16835def29e35d3fcef0b325b2b254ba08b7b"}
Sep 17 18:11:58.447808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.447735    2565 scope.go:117] "RemoveContainer" containerID="bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba"
Sep 17 18:11:58.455919 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.455897    2565 scope.go:117] "RemoveContainer" containerID="bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba"
Sep 17 18:11:58.456230 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:11:58.456207    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba\": container with ID starting with bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba not found: ID does not exist" containerID="bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba"
Sep 17 18:11:58.456278 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.456234    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba"} err="failed to get container status \"bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba\": rpc error: code = NotFound desc = could not find container \"bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba\": container with ID starting with bd80b41c776d9860b547632404161128d4a2af9e5feda9f9f3c2e348d78b31ba not found: ID does not exist"
Sep 17 18:11:58.468482 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.468455    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"]
Sep 17 18:11:58.473730 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.473712    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx"]
Sep 17 18:11:58.800870 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:58.800761    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9gxcx" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.41:9003\" within 1s: context deadline exceeded"
Sep 17 18:11:59.349935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:11:59.349902    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" path="/var/lib/kubelet/pods/b916ee05-c26f-4234-b50f-d0ff3232c5ac/volumes"
Sep 17 18:12:01.877858 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:01.877822    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"]
Sep 17 18:12:09.041372 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:09.041329    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:16.896890 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:16.896790    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerName="main" containerID="cri-o://c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2" gracePeriod=45
Sep 17 18:12:19.037209 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:19.037170    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:23.091638 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091602    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"]
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091882    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091893    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091914    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091920    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091927    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965f8420-ace8-4eec-8f10-bfe8a75e301d" containerName="main"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091933    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="965f8420-ace8-4eec-8f10-bfe8a75e301d" containerName="main"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091980    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="965f8420-ace8-4eec-8f10-bfe8a75e301d" containerName="main"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091989    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b916ee05-c26f-4234-b50f-d0ff3232c5ac" containerName="main"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.091997    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy"
Sep 17 18:12:23.092005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.092003    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy"
Sep 17 18:12:23.094673 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.094652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:23.097417 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.097378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:12:23.097564 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.097378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:12:23.097653 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.097421    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-qtjmg\""
Sep 17 18:12:23.097653 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.097479    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 18:12:23.097880 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.097517    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-xt8bz\""
Sep 17 18:12:23.105996 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.105974    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"]
Sep 17 18:12:23.206643 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.206605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ddaabc-82a7-4929-bab8-dae444552982-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5\" (UID: \"75ddaabc-82a7-4929-bab8-dae444552982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:23.206832 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.206715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqdm\" (UniqueName: \"kubernetes.io/projected/75ddaabc-82a7-4929-bab8-dae444552982-kube-api-access-tfqdm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5\" (UID: \"75ddaabc-82a7-4929-bab8-dae444552982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:23.307163 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.307127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ddaabc-82a7-4929-bab8-dae444552982-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5\" (UID: \"75ddaabc-82a7-4929-bab8-dae444552982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:23.307329 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.307196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqdm\" (UniqueName: \"kubernetes.io/projected/75ddaabc-82a7-4929-bab8-dae444552982-kube-api-access-tfqdm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5\" (UID: \"75ddaabc-82a7-4929-bab8-dae444552982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:23.309624 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.309595    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ddaabc-82a7-4929-bab8-dae444552982-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5\" (UID: \"75ddaabc-82a7-4929-bab8-dae444552982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:23.314568 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.314544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqdm\" (UniqueName: \"kubernetes.io/projected/75ddaabc-82a7-4929-bab8-dae444552982-kube-api-access-tfqdm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5\" (UID: \"75ddaabc-82a7-4929-bab8-dae444552982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:23.405543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.405476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:23.678827 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:23.678745    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"]
Sep 17 18:12:23.682533 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:12:23.682491    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ddaabc_82a7_4929_bab8_dae444552982.slice/crio-9f39dd060c010bf038f61b6428c7967ea360f24f634f416b418c2f7d9f43ba79 WatchSource:0}: Error finding container 9f39dd060c010bf038f61b6428c7967ea360f24f634f416b418c2f7d9f43ba79: Status 404 returned error can't find the container with id 9f39dd060c010bf038f61b6428c7967ea360f24f634f416b418c2f7d9f43ba79
Sep 17 18:12:24.540682 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:24.540645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" event={"ID":"75ddaabc-82a7-4929-bab8-dae444552982","Type":"ContainerStarted","Data":"8dc7f01a0d30f41a29c9b09e522ac4a90188609c5f7d0ac32b0f33a1c8372069"}
Sep 17 18:12:24.541065 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:24.540702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" event={"ID":"75ddaabc-82a7-4929-bab8-dae444552982","Type":"ContainerStarted","Data":"9f39dd060c010bf038f61b6428c7967ea360f24f634f416b418c2f7d9f43ba79"}
Sep 17 18:12:24.541065 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:24.540742    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:12:24.559961 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:24.559906    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" podStartSLOduration=1.559885228 podStartE2EDuration="1.559885228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:24.559308243 +0000 UTC m=+1887.395077465" watchObservedRunningTime="2026-09-17 18:12:24.559885228 +0000 UTC m=+1887.395654451"
Sep 17 18:12:29.037209 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:29.037172    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:29.037617 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:29.037266    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:12:39.038176 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:39.038131    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:47.136645 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.136618    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:12:47.297312 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.297231    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10270f87-242a-456f-a37b-7e8a6b6a5081-tls-certs\") pod \"10270f87-242a-456f-a37b-7e8a6b6a5081\" (UID: \"10270f87-242a-456f-a37b-7e8a6b6a5081\") "
Sep 17 18:12:47.297312 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.297280    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwg6j\" (UniqueName: \"kubernetes.io/projected/10270f87-242a-456f-a37b-7e8a6b6a5081-kube-api-access-xwg6j\") pod \"10270f87-242a-456f-a37b-7e8a6b6a5081\" (UID: \"10270f87-242a-456f-a37b-7e8a6b6a5081\") "
Sep 17 18:12:47.299485 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.299456    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10270f87-242a-456f-a37b-7e8a6b6a5081-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10270f87-242a-456f-a37b-7e8a6b6a5081" (UID: "10270f87-242a-456f-a37b-7e8a6b6a5081"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:47.299638 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.299616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10270f87-242a-456f-a37b-7e8a6b6a5081-kube-api-access-xwg6j" (OuterVolumeSpecName: "kube-api-access-xwg6j") pod "10270f87-242a-456f-a37b-7e8a6b6a5081" (UID: "10270f87-242a-456f-a37b-7e8a6b6a5081"). InnerVolumeSpecName "kube-api-access-xwg6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:47.398302 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.398279    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10270f87-242a-456f-a37b-7e8a6b6a5081-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:12:47.398302 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.398301    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwg6j\" (UniqueName: \"kubernetes.io/projected/10270f87-242a-456f-a37b-7e8a6b6a5081-kube-api-access-xwg6j\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:12:47.629084 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.628989    2565 generic.go:356] "Generic (PLEG): container finished" podID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerID="c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2" exitCode=0
Sep 17 18:12:47.629084 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.629031    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"
Sep 17 18:12:47.629084 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.629051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" event={"ID":"10270f87-242a-456f-a37b-7e8a6b6a5081","Type":"ContainerDied","Data":"c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2"}
Sep 17 18:12:47.629347 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.629099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4" event={"ID":"10270f87-242a-456f-a37b-7e8a6b6a5081","Type":"ContainerDied","Data":"f6d6e0ea27fd07b370821ce113017632a4a16f2d53cfc121ea8fa9a16c2a6f9e"}
Sep 17 18:12:47.629347 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.629123    2565 scope.go:117] "RemoveContainer" containerID="c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2"
Sep 17 18:12:47.639218 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.639189    2565 scope.go:117] "RemoveContainer" containerID="c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2"
Sep 17 18:12:47.639606 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:12:47.639585    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2\": container with ID starting with c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2 not found: ID does not exist" containerID="c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2"
Sep 17 18:12:47.639660 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.639614    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2"} err="failed to get container status \"c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2\": rpc error: code = NotFound desc = could not find container \"c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2\": container with ID starting with c3fc1b3d9e5830357dfa2e8cc8d526bb6aeb8f7eb3d09c8c44090edf955042c2 not found: ID does not exist"
Sep 17 18:12:47.644102 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.644080    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"]
Sep 17 18:12:47.647772 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:47.647746    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schehzxn4"]
Sep 17 18:12:49.348296 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:49.348262    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" path="/var/lib/kubelet/pods/10270f87-242a-456f-a37b-7e8a6b6a5081/volumes"
Sep 17 18:12:55.545730 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:12:55.545704    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:13:01.943684 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.943645    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"]
Sep 17 18:13:01.944144 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.944045    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerName="main"
Sep 17 18:13:01.944144 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.944065    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerName="main"
Sep 17 18:13:01.944144 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.944086    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy"
Sep 17 18:13:01.944144 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.944094    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c768b587-73b7-42e4-9f7a-3280904acc7e" containerName="istio-proxy"
Sep 17 18:13:01.944369 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.944150    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="10270f87-242a-456f-a37b-7e8a6b6a5081" containerName="main"
Sep 17 18:13:01.948380 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.948359    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:01.951264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.951239    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 18:13:01.951437 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.951240    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 18:13:01.951437 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.951244    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 18:13:01.951437 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.951245    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-xgtbn\""
Sep 17 18:13:01.951626 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.951275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-86sng\""
Sep 17 18:13:01.959166 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:01.959143    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"]
Sep 17 18:13:02.009997 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.009969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d\" (UID: \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:02.009997 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.009997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bgb\" (UniqueName: \"kubernetes.io/projected/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-kube-api-access-p6bgb\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d\" (UID: \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:02.111295 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.111257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d\" (UID: \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:02.111295 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.111293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bgb\" (UniqueName: \"kubernetes.io/projected/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-kube-api-access-p6bgb\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d\" (UID: \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:02.113800 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.113782    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d\" (UID: \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:02.119377 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.119358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bgb\" (UniqueName: \"kubernetes.io/projected/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-kube-api-access-p6bgb\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d\" (UID: \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:02.258452 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.258329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:02.527944 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.527865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"]
Sep 17 18:13:02.530728 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:13:02.530700    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26be0d95_bdbe_41f2_b2e4_8b4f1122c976.slice/crio-42c38d9c1ef22b8bba50175b8f28fefb2b1bdbad8f9397ee79e574f24cb1fda0 WatchSource:0}: Error finding container 42c38d9c1ef22b8bba50175b8f28fefb2b1bdbad8f9397ee79e574f24cb1fda0: Status 404 returned error can't find the container with id 42c38d9c1ef22b8bba50175b8f28fefb2b1bdbad8f9397ee79e574f24cb1fda0
Sep 17 18:13:02.684310 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.684278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" event={"ID":"26be0d95-bdbe-41f2-b2e4-8b4f1122c976","Type":"ContainerStarted","Data":"8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd"}
Sep 17 18:13:02.684514 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.684329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" event={"ID":"26be0d95-bdbe-41f2-b2e4-8b4f1122c976","Type":"ContainerStarted","Data":"42c38d9c1ef22b8bba50175b8f28fefb2b1bdbad8f9397ee79e574f24cb1fda0"}
Sep 17 18:13:02.684514 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.684374    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:13:02.701657 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:02.701609    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" podStartSLOduration=1.70159429 podStartE2EDuration="1.70159429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:02.700122157 +0000 UTC m=+1925.535891383" watchObservedRunningTime="2026-09-17 18:13:02.70159429 +0000 UTC m=+1925.537363552"
Sep 17 18:13:33.690283 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:13:33.690254    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:14:11.188515 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.188480    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"]
Sep 17 18:14:11.191761 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.191740    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.193909 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.193883    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 18:14:11.201252 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.201231    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"]
Sep 17 18:14:11.255687 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.255652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.255687 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.255687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmb9q\" (UniqueName: \"kubernetes.io/projected/3be0c25f-2bae-4470-afdd-5b6df03d6617-kube-api-access-pmb9q\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.255955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.255728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.255955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.255755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be0c25f-2bae-4470-afdd-5b6df03d6617-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.255955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.255771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.255955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.255848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.356668 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.356638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be0c25f-2bae-4470-afdd-5b6df03d6617-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.356668 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.356666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.356888 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.356701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.356888 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.356726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.356888 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.356742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmb9q\" (UniqueName: \"kubernetes.io/projected/3be0c25f-2bae-4470-afdd-5b6df03d6617-kube-api-access-pmb9q\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.356888 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.356767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.357149 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.357086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.357149 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.357139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.357262 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.357222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.359107 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.359089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.359311 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.359294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be0c25f-2bae-4470-afdd-5b6df03d6617-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.364205 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.364180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmb9q\" (UniqueName: \"kubernetes.io/projected/3be0c25f-2bae-4470-afdd-5b6df03d6617-kube-api-access-pmb9q\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.502689 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.502611    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:11.777867 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.777790    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"]
Sep 17 18:14:11.780847 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:14:11.780810    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be0c25f_2bae_4470_afdd_5b6df03d6617.slice/crio-e36886b48200ff129d7f0818efd8160c5dc237678662bcb1432b83145192e2ef WatchSource:0}: Error finding container e36886b48200ff129d7f0818efd8160c5dc237678662bcb1432b83145192e2ef: Status 404 returned error can't find the container with id e36886b48200ff129d7f0818efd8160c5dc237678662bcb1432b83145192e2ef
Sep 17 18:14:11.940136 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.940098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8" event={"ID":"3be0c25f-2bae-4470-afdd-5b6df03d6617","Type":"ContainerStarted","Data":"e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d"}
Sep 17 18:14:11.940341 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.940151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8" event={"ID":"3be0c25f-2bae-4470-afdd-5b6df03d6617","Type":"ContainerStarted","Data":"e36886b48200ff129d7f0818efd8160c5dc237678662bcb1432b83145192e2ef"}
Sep 17 18:14:11.959129 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:11.959079    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8" podStartSLOduration=0.959064074 podStartE2EDuration="959.064074ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:11.957582215 +0000 UTC m=+1994.793351438" watchObservedRunningTime="2026-09-17 18:14:11.959064074 +0000 UTC m=+1994.794833296"
Sep 17 18:14:12.503539 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:12.503509    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:21.503738 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:21.503688    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:21.515963 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:21.515938    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:21.990806 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:21.990777    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:45.353165 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.353133    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"]
Sep 17 18:14:45.362711 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:14:45.362674    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:14:45.362711 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:14:45.362711 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:14:45.362711 ip-10-0-129-186 kubenswrapper[2565]:  > containerID="e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d" cmd=["/bin/sleep","15"]
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.362726    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8" message="" err=<
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]:  >
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:14:45.362741    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8" podUID="3be0c25f-2bae-4470-afdd-5b6df03d6617" containerName="main" containerID="cri-o://e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d"
Sep 17 18:14:45.363012 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.362779    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8" podUID="3be0c25f-2bae-4470-afdd-5b6df03d6617" containerName="main" containerID="cri-o://e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d" gracePeriod=60
Sep 17 18:14:45.713573 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.713546    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:45.835988 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.835957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be0c25f-2bae-4470-afdd-5b6df03d6617-tls-certs\") pod \"3be0c25f-2bae-4470-afdd-5b6df03d6617\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") "
Sep 17 18:14:45.836180 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836022    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-model-cache\") pod \"3be0c25f-2bae-4470-afdd-5b6df03d6617\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") "
Sep 17 18:14:45.836180 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836041    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-dshm\") pod \"3be0c25f-2bae-4470-afdd-5b6df03d6617\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") "
Sep 17 18:14:45.836180 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836142    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-tmp-dir\") pod \"3be0c25f-2bae-4470-afdd-5b6df03d6617\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") "
Sep 17 18:14:45.836357 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmb9q\" (UniqueName: \"kubernetes.io/projected/3be0c25f-2bae-4470-afdd-5b6df03d6617-kube-api-access-pmb9q\") pod \"3be0c25f-2bae-4470-afdd-5b6df03d6617\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") "
Sep 17 18:14:45.836357 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-home\") pod \"3be0c25f-2bae-4470-afdd-5b6df03d6617\" (UID: \"3be0c25f-2bae-4470-afdd-5b6df03d6617\") "
Sep 17 18:14:45.836357 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836262    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-model-cache" (OuterVolumeSpecName: "model-cache") pod "3be0c25f-2bae-4470-afdd-5b6df03d6617" (UID: "3be0c25f-2bae-4470-afdd-5b6df03d6617"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:45.836509 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836360    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3be0c25f-2bae-4470-afdd-5b6df03d6617" (UID: "3be0c25f-2bae-4470-afdd-5b6df03d6617"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:45.836509 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836494    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-home" (OuterVolumeSpecName: "home") pod "3be0c25f-2bae-4470-afdd-5b6df03d6617" (UID: "3be0c25f-2bae-4470-afdd-5b6df03d6617"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:45.836590 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836557    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:14:45.836626 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836598    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:14:45.836626 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.836612    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:14:45.838378 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.838347    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be0c25f-2bae-4470-afdd-5b6df03d6617-kube-api-access-pmb9q" (OuterVolumeSpecName: "kube-api-access-pmb9q") pod "3be0c25f-2bae-4470-afdd-5b6df03d6617" (UID: "3be0c25f-2bae-4470-afdd-5b6df03d6617"). InnerVolumeSpecName "kube-api-access-pmb9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:45.838378 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.838368    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be0c25f-2bae-4470-afdd-5b6df03d6617-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3be0c25f-2bae-4470-afdd-5b6df03d6617" (UID: "3be0c25f-2bae-4470-afdd-5b6df03d6617"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:45.838533 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.838386    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-dshm" (OuterVolumeSpecName: "dshm") pod "3be0c25f-2bae-4470-afdd-5b6df03d6617" (UID: "3be0c25f-2bae-4470-afdd-5b6df03d6617"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:45.937699 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.937660    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmb9q\" (UniqueName: \"kubernetes.io/projected/3be0c25f-2bae-4470-afdd-5b6df03d6617-kube-api-access-pmb9q\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:14:45.937699 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.937698    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be0c25f-2bae-4470-afdd-5b6df03d6617-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:14:45.937887 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:45.937712    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be0c25f-2bae-4470-afdd-5b6df03d6617-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:14:46.070029 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.069993    2565 generic.go:356] "Generic (PLEG): container finished" podID="3be0c25f-2bae-4470-afdd-5b6df03d6617" containerID="e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d" exitCode=0
Sep 17 18:14:46.070224 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.070039    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"
Sep 17 18:14:46.070224 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.070058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8" event={"ID":"3be0c25f-2bae-4470-afdd-5b6df03d6617","Type":"ContainerDied","Data":"e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d"}
Sep 17 18:14:46.070224 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.070099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8" event={"ID":"3be0c25f-2bae-4470-afdd-5b6df03d6617","Type":"ContainerDied","Data":"e36886b48200ff129d7f0818efd8160c5dc237678662bcb1432b83145192e2ef"}
Sep 17 18:14:46.070224 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.070120    2565 scope.go:117] "RemoveContainer" containerID="e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d"
Sep 17 18:14:46.079212 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.079193    2565 scope.go:117] "RemoveContainer" containerID="e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d"
Sep 17 18:14:46.079485 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:14:46.079467    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d\": container with ID starting with e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d not found: ID does not exist" containerID="e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d"
Sep 17 18:14:46.079548 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.079493    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d"} err="failed to get container status \"e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d\": rpc error: code = NotFound desc = could not find container \"e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d\": container with ID starting with e02e493fcfe78e00b6517f62c84aed3f2a3e3c4bb7c912adf10c17d1af4ecd4d not found: ID does not exist"
Sep 17 18:14:46.090130 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.090107    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"]
Sep 17 18:14:46.092976 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:46.092952    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-jbkb8"]
Sep 17 18:14:47.348535 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:14:47.348501    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3be0c25f-2bae-4470-afdd-5b6df03d6617" path="/var/lib/kubelet/pods/3be0c25f-2bae-4470-afdd-5b6df03d6617/volumes"
Sep 17 18:15:06.226703 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:06.226666    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"]
Sep 17 18:15:13.690050 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:13.689963    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:21.245617 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:21.245554    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerName="main" containerID="cri-o://8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd" gracePeriod=45
Sep 17 18:15:23.689640 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:23.689605    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:33.690256 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:33.690220    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:33.690648 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:33.690347    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:15:35.201504 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:35.201474    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"]
Sep 17 18:15:35.546183 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:35.546098    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:43.689956 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:43.689916    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:45.545939 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:45.545903    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:50.221894 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:50.221828    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main" containerID="cri-o://8dc7f01a0d30f41a29c9b09e522ac4a90188609c5f7d0ac32b0f33a1c8372069" gracePeriod=45
Sep 17 18:15:51.485765 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:51.485738    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:15:51.513648 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:51.513592    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-tls-certs\") pod \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\" (UID: \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\") "
Sep 17 18:15:51.513648 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:51.513645    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6bgb\" (UniqueName: \"kubernetes.io/projected/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-kube-api-access-p6bgb\") pod \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\" (UID: \"26be0d95-bdbe-41f2-b2e4-8b4f1122c976\") "
Sep 17 18:15:51.515814 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:51.515775    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26be0d95-bdbe-41f2-b2e4-8b4f1122c976" (UID: "26be0d95-bdbe-41f2-b2e4-8b4f1122c976"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:51.515940 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:51.515901    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-kube-api-access-p6bgb" (OuterVolumeSpecName: "kube-api-access-p6bgb") pod "26be0d95-bdbe-41f2-b2e4-8b4f1122c976" (UID: "26be0d95-bdbe-41f2-b2e4-8b4f1122c976"). InnerVolumeSpecName "kube-api-access-p6bgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:51.614478 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:51.614443    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6bgb\" (UniqueName: \"kubernetes.io/projected/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-kube-api-access-p6bgb\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:15:51.614478 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:51.614472    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be0d95-bdbe-41f2-b2e4-8b4f1122c976-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:15:52.312782 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.312742    2565 generic.go:356] "Generic (PLEG): container finished" podID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerID="8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd" exitCode=0
Sep 17 18:15:52.312950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.312789    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"
Sep 17 18:15:52.312950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.312803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" event={"ID":"26be0d95-bdbe-41f2-b2e4-8b4f1122c976","Type":"ContainerDied","Data":"8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd"}
Sep 17 18:15:52.312950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.312855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d" event={"ID":"26be0d95-bdbe-41f2-b2e4-8b4f1122c976","Type":"ContainerDied","Data":"42c38d9c1ef22b8bba50175b8f28fefb2b1bdbad8f9397ee79e574f24cb1fda0"}
Sep 17 18:15:52.312950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.312884    2565 scope.go:117] "RemoveContainer" containerID="8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd"
Sep 17 18:15:52.323962 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.323947    2565 scope.go:117] "RemoveContainer" containerID="8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd"
Sep 17 18:15:52.324203 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:15:52.324186    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd\": container with ID starting with 8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd not found: ID does not exist" containerID="8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd"
Sep 17 18:15:52.324241 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.324212    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd"} err="failed to get container status \"8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd\": rpc error: code = NotFound desc = could not find container \"8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd\": container with ID starting with 8bf5298c8543dd880770a24d6b725124b7b3e439b0196f75e3078bca0c197dcd not found: ID does not exist"
Sep 17 18:15:52.333515 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.333491    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"]
Sep 17 18:15:52.339938 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:52.339914    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche29l7d"]
Sep 17 18:15:53.349036 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:53.349001    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" path="/var/lib/kubelet/pods/26be0d95-bdbe-41f2-b2e4-8b4f1122c976/volumes"
Sep 17 18:15:55.545648 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:55.545612    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:55.546026 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:55.545720    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:15:59.400457 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:59.400434    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:15:59.403412 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:15:59.403374    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:16:05.545693 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:05.545656    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:16:11.761084 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.761049    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"]
Sep 17 18:16:11.761545 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.761439    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerName="main"
Sep 17 18:16:11.761545 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.761459    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerName="main"
Sep 17 18:16:11.761545 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.761480    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be0c25f-2bae-4470-afdd-5b6df03d6617" containerName="main"
Sep 17 18:16:11.761545 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.761486    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be0c25f-2bae-4470-afdd-5b6df03d6617" containerName="main"
Sep 17 18:16:11.761831 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.761549    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be0c25f-2bae-4470-afdd-5b6df03d6617" containerName="main"
Sep 17 18:16:11.761831 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.761561    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="26be0d95-bdbe-41f2-b2e4-8b4f1122c976" containerName="main"
Sep 17 18:16:11.767028 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.767005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:11.769488 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.769463    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-x7tt2\""
Sep 17 18:16:11.770263 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.770243    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 18:16:11.770336 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.770245    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 18:16:11.770336 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.770283    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-2mv88\""
Sep 17 18:16:11.770565 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.770552    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 18:16:11.774734 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.774712    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"]
Sep 17 18:16:11.868141 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.868105    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2bl\" (UniqueName: \"kubernetes.io/projected/38904514-70b4-4587-b94c-b0d8379e560e-kube-api-access-zg2bl\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv\" (UID: \"38904514-70b4-4587-b94c-b0d8379e560e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:11.868314 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.868148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38904514-70b4-4587-b94c-b0d8379e560e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv\" (UID: \"38904514-70b4-4587-b94c-b0d8379e560e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:11.969521 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.969486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2bl\" (UniqueName: \"kubernetes.io/projected/38904514-70b4-4587-b94c-b0d8379e560e-kube-api-access-zg2bl\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv\" (UID: \"38904514-70b4-4587-b94c-b0d8379e560e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:11.969521 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.969525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38904514-70b4-4587-b94c-b0d8379e560e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv\" (UID: \"38904514-70b4-4587-b94c-b0d8379e560e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:11.971933 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.971911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38904514-70b4-4587-b94c-b0d8379e560e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv\" (UID: \"38904514-70b4-4587-b94c-b0d8379e560e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:11.977535 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:11.977512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2bl\" (UniqueName: \"kubernetes.io/projected/38904514-70b4-4587-b94c-b0d8379e560e-kube-api-access-zg2bl\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv\" (UID: \"38904514-70b4-4587-b94c-b0d8379e560e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:12.077562 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:12.077484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:12.355726 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:12.355645    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"]
Sep 17 18:16:12.358467 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:16:12.358443    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38904514_70b4_4587_b94c_b0d8379e560e.slice/crio-c5982d23844e682a869cd96a03c77795420c986cefca086af1fd27a2b0d2e5ac WatchSource:0}: Error finding container c5982d23844e682a869cd96a03c77795420c986cefca086af1fd27a2b0d2e5ac: Status 404 returned error can't find the container with id c5982d23844e682a869cd96a03c77795420c986cefca086af1fd27a2b0d2e5ac
Sep 17 18:16:12.393151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:12.393122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" event={"ID":"38904514-70b4-4587-b94c-b0d8379e560e","Type":"ContainerStarted","Data":"c5982d23844e682a869cd96a03c77795420c986cefca086af1fd27a2b0d2e5ac"}
Sep 17 18:16:13.398610 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:13.398570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" event={"ID":"38904514-70b4-4587-b94c-b0d8379e560e","Type":"ContainerStarted","Data":"b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2"}
Sep 17 18:16:13.399015 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:13.398686    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:16:13.415147 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:13.415102    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" podStartSLOduration=2.415089203 podStartE2EDuration="2.415089203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:13.413943984 +0000 UTC m=+2116.249713208" watchObservedRunningTime="2026-09-17 18:16:13.415089203 +0000 UTC m=+2116.250858443"
Sep 17 18:16:15.545389 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:15.545353    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:16:20.427062 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.427030    2565 generic.go:356] "Generic (PLEG): container finished" podID="75ddaabc-82a7-4929-bab8-dae444552982" containerID="8dc7f01a0d30f41a29c9b09e522ac4a90188609c5f7d0ac32b0f33a1c8372069" exitCode=0
Sep 17 18:16:20.427443 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.427080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" event={"ID":"75ddaabc-82a7-4929-bab8-dae444552982","Type":"ContainerDied","Data":"8dc7f01a0d30f41a29c9b09e522ac4a90188609c5f7d0ac32b0f33a1c8372069"}
Sep 17 18:16:20.467085 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.467059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:16:20.540313 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.540234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ddaabc-82a7-4929-bab8-dae444552982-tls-certs\") pod \"75ddaabc-82a7-4929-bab8-dae444552982\" (UID: \"75ddaabc-82a7-4929-bab8-dae444552982\") "
Sep 17 18:16:20.540484 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.540317    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfqdm\" (UniqueName: \"kubernetes.io/projected/75ddaabc-82a7-4929-bab8-dae444552982-kube-api-access-tfqdm\") pod \"75ddaabc-82a7-4929-bab8-dae444552982\" (UID: \"75ddaabc-82a7-4929-bab8-dae444552982\") "
Sep 17 18:16:20.542364 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.542327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ddaabc-82a7-4929-bab8-dae444552982-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75ddaabc-82a7-4929-bab8-dae444552982" (UID: "75ddaabc-82a7-4929-bab8-dae444552982"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:20.542498 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.542372    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ddaabc-82a7-4929-bab8-dae444552982-kube-api-access-tfqdm" (OuterVolumeSpecName: "kube-api-access-tfqdm") pod "75ddaabc-82a7-4929-bab8-dae444552982" (UID: "75ddaabc-82a7-4929-bab8-dae444552982"). InnerVolumeSpecName "kube-api-access-tfqdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:20.641793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.641759    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ddaabc-82a7-4929-bab8-dae444552982-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:16:20.641793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:20.641789    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqdm\" (UniqueName: \"kubernetes.io/projected/75ddaabc-82a7-4929-bab8-dae444552982-kube-api-access-tfqdm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:16:21.431523 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:21.431491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5" event={"ID":"75ddaabc-82a7-4929-bab8-dae444552982","Type":"ContainerDied","Data":"9f39dd060c010bf038f61b6428c7967ea360f24f634f416b418c2f7d9f43ba79"}
Sep 17 18:16:21.431903 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:21.431524    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"
Sep 17 18:16:21.431903 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:21.431548    2565 scope.go:117] "RemoveContainer" containerID="8dc7f01a0d30f41a29c9b09e522ac4a90188609c5f7d0ac32b0f33a1c8372069"
Sep 17 18:16:21.445364 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:21.445341    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"]
Sep 17 18:16:21.448551 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:21.448525    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schej9kv5"]
Sep 17 18:16:23.348378 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:23.348342    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ddaabc-82a7-4929-bab8-dae444552982" path="/var/lib/kubelet/pods/75ddaabc-82a7-4929-bab8-dae444552982/volumes"
Sep 17 18:16:44.404280 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:16:44.404195    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:18:26.510471 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:18:26.510442    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"]
Sep 17 18:18:34.403710 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:18:34.403673    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" podUID="38904514-70b4-4587-b94c-b0d8379e560e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:41.532418 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:18:41.532362    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" podUID="38904514-70b4-4587-b94c-b0d8379e560e" containerName="main" containerID="cri-o://b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2" gracePeriod=45
Sep 17 18:18:44.403982 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:18:44.403937    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" podUID="38904514-70b4-4587-b94c-b0d8379e560e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:54.404491 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:18:54.404449    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" podUID="38904514-70b4-4587-b94c-b0d8379e560e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:54.404874 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:18:54.404548    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:19:04.404435 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:04.404373    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" podUID="38904514-70b4-4587-b94c-b0d8379e560e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:11.777210 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:11.777181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:19:11.926688 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:11.926653    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg2bl\" (UniqueName: \"kubernetes.io/projected/38904514-70b4-4587-b94c-b0d8379e560e-kube-api-access-zg2bl\") pod \"38904514-70b4-4587-b94c-b0d8379e560e\" (UID: \"38904514-70b4-4587-b94c-b0d8379e560e\") "
Sep 17 18:19:11.926878 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:11.926700    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38904514-70b4-4587-b94c-b0d8379e560e-tls-certs\") pod \"38904514-70b4-4587-b94c-b0d8379e560e\" (UID: \"38904514-70b4-4587-b94c-b0d8379e560e\") "
Sep 17 18:19:11.928898 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:11.928862    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38904514-70b4-4587-b94c-b0d8379e560e-kube-api-access-zg2bl" (OuterVolumeSpecName: "kube-api-access-zg2bl") pod "38904514-70b4-4587-b94c-b0d8379e560e" (UID: "38904514-70b4-4587-b94c-b0d8379e560e"). InnerVolumeSpecName "kube-api-access-zg2bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:11.929009 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:11.928944    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38904514-70b4-4587-b94c-b0d8379e560e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38904514-70b4-4587-b94c-b0d8379e560e" (UID: "38904514-70b4-4587-b94c-b0d8379e560e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:12.027527 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.027489    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg2bl\" (UniqueName: \"kubernetes.io/projected/38904514-70b4-4587-b94c-b0d8379e560e-kube-api-access-zg2bl\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:19:12.027527 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.027520    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38904514-70b4-4587-b94c-b0d8379e560e-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:19:12.043584 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.043556    2565 generic.go:356] "Generic (PLEG): container finished" podID="38904514-70b4-4587-b94c-b0d8379e560e" containerID="b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2" exitCode=0
Sep 17 18:19:12.043721 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.043602    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"
Sep 17 18:19:12.043721 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.043613    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" event={"ID":"38904514-70b4-4587-b94c-b0d8379e560e","Type":"ContainerDied","Data":"b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2"}
Sep 17 18:19:12.043721 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.043654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv" event={"ID":"38904514-70b4-4587-b94c-b0d8379e560e","Type":"ContainerDied","Data":"c5982d23844e682a869cd96a03c77795420c986cefca086af1fd27a2b0d2e5ac"}
Sep 17 18:19:12.043721 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.043675    2565 scope.go:117] "RemoveContainer" containerID="b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2"
Sep 17 18:19:12.052481 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.052461    2565 scope.go:117] "RemoveContainer" containerID="b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2"
Sep 17 18:19:12.052806 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:19:12.052784    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2\": container with ID starting with b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2 not found: ID does not exist" containerID="b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2"
Sep 17 18:19:12.052876 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.052821    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2"} err="failed to get container status \"b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2\": rpc error: code = NotFound desc = could not find container \"b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2\": container with ID starting with b823159e4cbf618e98ece896ffe643ed298931b44cf49767abb88c2e965fdab2 not found: ID does not exist"
Sep 17 18:19:12.064029 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.064006    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"]
Sep 17 18:19:12.067409 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:12.067377    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58x7qwv"]
Sep 17 18:19:13.348744 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:13.348710    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38904514-70b4-4587-b94c-b0d8379e560e" path="/var/lib/kubelet/pods/38904514-70b4-4587-b94c-b0d8379e560e/volumes"
Sep 17 18:19:16.820068 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.820037    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"]
Sep 17 18:19:16.820463 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.820299    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main"
Sep 17 18:19:16.820463 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.820310    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main"
Sep 17 18:19:16.820463 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.820319    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38904514-70b4-4587-b94c-b0d8379e560e" containerName="main"
Sep 17 18:19:16.820463 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.820324    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38904514-70b4-4587-b94c-b0d8379e560e" containerName="main"
Sep 17 18:19:16.820463 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.820363    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ddaabc-82a7-4929-bab8-dae444552982" containerName="main"
Sep 17 18:19:16.820463 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.820371    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38904514-70b4-4587-b94c-b0d8379e560e" containerName="main"
Sep 17 18:19:16.824610 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.824590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.827006 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.826937    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 18:19:16.827006 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.826994    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:19:16.827215 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.826999    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-828kr\""
Sep 17 18:19:16.827215 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.827037    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 18:19:16.835153 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.835129    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"]
Sep 17 18:19:16.963533 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.963533 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86967078-0345-4bf3-919a-8d874c0268f1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.963746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.963746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.963746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963664    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.963746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.963880 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86967078-0345-4bf3-919a-8d874c0268f1-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.963880 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:16.963880 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:16.963789    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqg7f\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-kube-api-access-zqg7f\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065090 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065090 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86967078-0345-4bf3-919a-8d874c0268f1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065326 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065113    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065326 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065326 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065326 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065326 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86967078-0345-4bf3-919a-8d874c0268f1-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065604 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065673 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065732 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065732 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065845 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqg7f\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-kube-api-access-zqg7f\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065845 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.065988 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.065964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86967078-0345-4bf3-919a-8d874c0268f1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.067705 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.067681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.067852 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.067835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86967078-0345-4bf3-919a-8d874c0268f1-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.073753 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.073690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.074068 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.074049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqg7f\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-kube-api-access-zqg7f\") pod \"router-gateway-1-openshift-default-cc7cb45dd-d4spx\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.135880 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.135856    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:17.417670 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.417638    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"]
Sep 17 18:19:17.420742 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:19:17.420713    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86967078_0345_4bf3_919a_8d874c0268f1.slice/crio-c86de2485e699ed71cdda0e7fd6b99169b4bbe8d8dcc119e0a00bdd785b9d3d3 WatchSource:0}: Error finding container c86de2485e699ed71cdda0e7fd6b99169b4bbe8d8dcc119e0a00bdd785b9d3d3: Status 404 returned error can't find the container with id c86de2485e699ed71cdda0e7fd6b99169b4bbe8d8dcc119e0a00bdd785b9d3d3
Sep 17 18:19:17.422863 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.422829    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:19:17.422971 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.422932    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:19:17.423035 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:17.422993    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:19:18.066706 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:18.066668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" event={"ID":"86967078-0345-4bf3-919a-8d874c0268f1","Type":"ContainerStarted","Data":"247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75"}
Sep 17 18:19:18.067084 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:18.066722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" event={"ID":"86967078-0345-4bf3-919a-8d874c0268f1","Type":"ContainerStarted","Data":"c86de2485e699ed71cdda0e7fd6b99169b4bbe8d8dcc119e0a00bdd785b9d3d3"}
Sep 17 18:19:18.086123 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:18.086071    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" podStartSLOduration=2.086056058 podStartE2EDuration="2.086056058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:18.085019184 +0000 UTC m=+2300.920788407" watchObservedRunningTime="2026-09-17 18:19:18.086056058 +0000 UTC m=+2300.921825281"
Sep 17 18:19:18.136361 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:18.136332    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:18.141430 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:18.141402    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:19.070554 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:19.070517    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:19.071343 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:19.071325    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:29.112186 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:29.112102    2565 generic.go:356] "Generic (PLEG): container finished" podID="86967078-0345-4bf3-919a-8d874c0268f1" containerID="247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75" exitCode=0
Sep 17 18:19:29.112186 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:29.112138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" event={"ID":"86967078-0345-4bf3-919a-8d874c0268f1","Type":"ContainerDied","Data":"247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75"}
Sep 17 18:19:29.112712 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:29.112542    2565 scope.go:117] "RemoveContainer" containerID="247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75"
Sep 17 18:19:29.113433 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:29.113385    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:19:29.113506 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:29.113487    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:19:29.113543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:29.113533    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:19:30.117175 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:30.117145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" event={"ID":"86967078-0345-4bf3-919a-8d874c0268f1","Type":"ContainerStarted","Data":"8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa"}
Sep 17 18:19:30.136785 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:30.136757    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:30.137901 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:30.137873    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 17 18:19:31.136642 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:31.136605    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 17 18:19:32.137035 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:32.136996    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 17 18:19:33.136443 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:33.136387    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 17 18:19:34.070865 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:34.070820    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:34.136135 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:34.136098    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 17 18:19:35.140301 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:35.140268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:35.141364 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:35.141342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:19:40.748921 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.748886    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"]
Sep 17 18:19:40.751459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.751440    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:40.753713 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.753690    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 18:19:40.754201 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.754162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-fjj4v\""
Sep 17 18:19:40.754299 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.754158    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 18:19:40.754299 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.754236    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 18:19:40.754498 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.754480    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-rk9dn\""
Sep 17 18:19:40.761149 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.761128    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"]
Sep 17 18:19:40.854968 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.854917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db655096-b58c-4633-a15d-34e68bc23879-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b\" (UID: \"db655096-b58c-4633-a15d-34e68bc23879\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:40.855154 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.855012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48z7\" (UniqueName: \"kubernetes.io/projected/db655096-b58c-4633-a15d-34e68bc23879-kube-api-access-x48z7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b\" (UID: \"db655096-b58c-4633-a15d-34e68bc23879\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:40.956128 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.956055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48z7\" (UniqueName: \"kubernetes.io/projected/db655096-b58c-4633-a15d-34e68bc23879-kube-api-access-x48z7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b\" (UID: \"db655096-b58c-4633-a15d-34e68bc23879\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:40.956277 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.956125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db655096-b58c-4633-a15d-34e68bc23879-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b\" (UID: \"db655096-b58c-4633-a15d-34e68bc23879\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:40.958678 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.958655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db655096-b58c-4633-a15d-34e68bc23879-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b\" (UID: \"db655096-b58c-4633-a15d-34e68bc23879\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:40.965070 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:40.965048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48z7\" (UniqueName: \"kubernetes.io/projected/db655096-b58c-4633-a15d-34e68bc23879-kube-api-access-x48z7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b\" (UID: \"db655096-b58c-4633-a15d-34e68bc23879\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:41.062694 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:41.062632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:41.331808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:41.331618    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"]
Sep 17 18:19:41.334596 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:19:41.334546    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb655096_b58c_4633_a15d_34e68bc23879.slice/crio-8d5092144778ace1ecc4cce469e3e3e64a1de5e5b1842c08516372e53525e8d4 WatchSource:0}: Error finding container 8d5092144778ace1ecc4cce469e3e3e64a1de5e5b1842c08516372e53525e8d4: Status 404 returned error can't find the container with id 8d5092144778ace1ecc4cce469e3e3e64a1de5e5b1842c08516372e53525e8d4
Sep 17 18:19:42.165045 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:42.164999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" event={"ID":"db655096-b58c-4633-a15d-34e68bc23879","Type":"ContainerStarted","Data":"216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8"}
Sep 17 18:19:42.165576 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:42.165071    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" event={"ID":"db655096-b58c-4633-a15d-34e68bc23879","Type":"ContainerStarted","Data":"8d5092144778ace1ecc4cce469e3e3e64a1de5e5b1842c08516372e53525e8d4"}
Sep 17 18:19:42.165913 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:42.165883    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:19:42.181914 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:19:42.181863    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" podStartSLOduration=2.181847648 podStartE2EDuration="2.181847648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:42.180275773 +0000 UTC m=+2325.016044995" watchObservedRunningTime="2026-09-17 18:19:42.181847648 +0000 UTC m=+2325.017616871"
Sep 17 18:20:14.175374 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:20:14.175342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:20:59.422467 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:20:59.422437    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:20:59.426387 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:20:59.426364    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:21:02.607294 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:02.607256    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"]
Sep 17 18:21:04.174784 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:04.174752    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:14.174730 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:14.174634    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:17.627944 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:17.627875    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main" containerID="cri-o://216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8" gracePeriod=45
Sep 17 18:21:24.174843 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:24.174804    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:24.175296 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:24.174912    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:21:34.174941 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:34.174900    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:44.175372 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:44.175333    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:47.865268 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:47.865242    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:21:47.948872 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:47.948841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db655096-b58c-4633-a15d-34e68bc23879-tls-certs\") pod \"db655096-b58c-4633-a15d-34e68bc23879\" (UID: \"db655096-b58c-4633-a15d-34e68bc23879\") "
Sep 17 18:21:47.949039 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:47.948887    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x48z7\" (UniqueName: \"kubernetes.io/projected/db655096-b58c-4633-a15d-34e68bc23879-kube-api-access-x48z7\") pod \"db655096-b58c-4633-a15d-34e68bc23879\" (UID: \"db655096-b58c-4633-a15d-34e68bc23879\") "
Sep 17 18:21:47.951109 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:47.951076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db655096-b58c-4633-a15d-34e68bc23879-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db655096-b58c-4633-a15d-34e68bc23879" (UID: "db655096-b58c-4633-a15d-34e68bc23879"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:47.951109 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:47.951092    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db655096-b58c-4633-a15d-34e68bc23879-kube-api-access-x48z7" (OuterVolumeSpecName: "kube-api-access-x48z7") pod "db655096-b58c-4633-a15d-34e68bc23879" (UID: "db655096-b58c-4633-a15d-34e68bc23879"). InnerVolumeSpecName "kube-api-access-x48z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:48.049904 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.049870    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db655096-b58c-4633-a15d-34e68bc23879-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:21:48.049904 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.049899    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48z7\" (UniqueName: \"kubernetes.io/projected/db655096-b58c-4633-a15d-34e68bc23879-kube-api-access-x48z7\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:21:48.630225 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.630188    2565 generic.go:356] "Generic (PLEG): container finished" podID="db655096-b58c-4633-a15d-34e68bc23879" containerID="216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8" exitCode=0
Sep 17 18:21:48.630417 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.630232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" event={"ID":"db655096-b58c-4633-a15d-34e68bc23879","Type":"ContainerDied","Data":"216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8"}
Sep 17 18:21:48.630417 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.630274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b" event={"ID":"db655096-b58c-4633-a15d-34e68bc23879","Type":"ContainerDied","Data":"8d5092144778ace1ecc4cce469e3e3e64a1de5e5b1842c08516372e53525e8d4"}
Sep 17 18:21:48.630417 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.630294    2565 scope.go:117] "RemoveContainer" containerID="216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8"
Sep 17 18:21:48.630417 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.630237    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"
Sep 17 18:21:48.638570 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.638549    2565 scope.go:117] "RemoveContainer" containerID="216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8"
Sep 17 18:21:48.638819 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:21:48.638801    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8\": container with ID starting with 216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8 not found: ID does not exist" containerID="216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8"
Sep 17 18:21:48.638881 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.638826    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8"} err="failed to get container status \"216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8\": rpc error: code = NotFound desc = could not find container \"216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8\": container with ID starting with 216fea0f72b573e9cd5aad4092aa6863411d84d1922b5a300b6c8f347853a3a8 not found: ID does not exist"
Sep 17 18:21:48.650219 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.650192    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"]
Sep 17 18:21:48.655061 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:48.655037    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schec5k5b"]
Sep 17 18:21:49.349026 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:21:49.348994    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db655096-b58c-4633-a15d-34e68bc23879" path="/var/lib/kubelet/pods/db655096-b58c-4633-a15d-34e68bc23879/volumes"
Sep 17 18:22:09.610182 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.610149    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"]
Sep 17 18:22:09.610836 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.610510    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main"
Sep 17 18:22:09.610836 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.610529    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main"
Sep 17 18:22:09.610836 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.610602    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db655096-b58c-4633-a15d-34e68bc23879" containerName="main"
Sep 17 18:22:09.613279 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.613263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.615183 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.615156    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:09.615322 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.615186    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 18:22:09.615611 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.615590    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 18:22:09.615715 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.615618    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 18:22:09.615715 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.615673    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 18:22:09.615845 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.615771    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-vw7gx\""
Sep 17 18:22:09.625119 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.625098    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"]
Sep 17 18:22:09.725904 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.725879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-dshm\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.726050 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.725911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-model-cache\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.726050 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.725929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da45b795-a647-476d-87bc-7d344102bbde-tls-certs\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.726050 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.725960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fvl\" (UniqueName: \"kubernetes.io/projected/da45b795-a647-476d-87bc-7d344102bbde-kube-api-access-47fvl\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.726050 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.726013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-home\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.726191 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.726090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-tmp-dir\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.726191 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.726117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-kserve-provision-location\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.726191 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.726149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da45b795-a647-476d-87bc-7d344102bbde-cabundle-cert\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827403 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-tmp-dir\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827558 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-kserve-provision-location\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827558 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827456    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da45b795-a647-476d-87bc-7d344102bbde-cabundle-cert\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827558 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-dshm\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827558 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-model-cache\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827792 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da45b795-a647-476d-87bc-7d344102bbde-tls-certs\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827792 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fvl\" (UniqueName: \"kubernetes.io/projected/da45b795-a647-476d-87bc-7d344102bbde-kube-api-access-47fvl\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827792 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-home\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827956 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-tmp-dir\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.827956 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.827856    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-model-cache\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.828098 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.828026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-kserve-provision-location\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.828098 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.828077    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-home\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.828268 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.828251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da45b795-a647-476d-87bc-7d344102bbde-cabundle-cert\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.829762 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.829746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-dshm\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.830098 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.830080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da45b795-a647-476d-87bc-7d344102bbde-tls-certs\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.835504 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.835484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fvl\" (UniqueName: \"kubernetes.io/projected/da45b795-a647-476d-87bc-7d344102bbde-kube-api-access-47fvl\") pod \"stop-feature-test-kserve-75fff8659b-x2kkf\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:09.884897 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.884837    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"]
Sep 17 18:22:09.887586 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.887567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:09.889506 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.889484    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-qd5m6\""
Sep 17 18:22:09.899643 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.899617    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"]
Sep 17 18:22:09.923499 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:09.923480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:10.029184 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.029149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cffed755-9673-47a1-9c9b-c79e7c93d911-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7\" (UID: \"cffed755-9673-47a1-9c9b-c79e7c93d911\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:10.029346 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.029203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db5ww\" (UniqueName: \"kubernetes.io/projected/cffed755-9673-47a1-9c9b-c79e7c93d911-kube-api-access-db5ww\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7\" (UID: \"cffed755-9673-47a1-9c9b-c79e7c93d911\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:10.129896 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.129864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cffed755-9673-47a1-9c9b-c79e7c93d911-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7\" (UID: \"cffed755-9673-47a1-9c9b-c79e7c93d911\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:10.130057 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.129916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db5ww\" (UniqueName: \"kubernetes.io/projected/cffed755-9673-47a1-9c9b-c79e7c93d911-kube-api-access-db5ww\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7\" (UID: \"cffed755-9673-47a1-9c9b-c79e7c93d911\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:10.132388 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.132367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cffed755-9673-47a1-9c9b-c79e7c93d911-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7\" (UID: \"cffed755-9673-47a1-9c9b-c79e7c93d911\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:10.137517 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.137443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db5ww\" (UniqueName: \"kubernetes.io/projected/cffed755-9673-47a1-9c9b-c79e7c93d911-kube-api-access-db5ww\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7\" (UID: \"cffed755-9673-47a1-9c9b-c79e7c93d911\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:10.196227 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.196188    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"]
Sep 17 18:22:10.198251 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:22:10.198224    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda45b795_a647_476d_87bc_7d344102bbde.slice/crio-bdc1e074787da40a1fc375d3c35966f3d949508c76c38cbdb21c19315e6b95ee WatchSource:0}: Error finding container bdc1e074787da40a1fc375d3c35966f3d949508c76c38cbdb21c19315e6b95ee: Status 404 returned error can't find the container with id bdc1e074787da40a1fc375d3c35966f3d949508c76c38cbdb21c19315e6b95ee
Sep 17 18:22:10.199998 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.199975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:10.493823 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.493790    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"]
Sep 17 18:22:10.497383 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:22:10.497353    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffed755_9673_47a1_9c9b_c79e7c93d911.slice/crio-3c31abe4e43ac79088d7f587da64752409439080e52cc9de3a869c98ce754a2c WatchSource:0}: Error finding container 3c31abe4e43ac79088d7f587da64752409439080e52cc9de3a869c98ce754a2c: Status 404 returned error can't find the container with id 3c31abe4e43ac79088d7f587da64752409439080e52cc9de3a869c98ce754a2c
Sep 17 18:22:10.709873 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.709834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" event={"ID":"cffed755-9673-47a1-9c9b-c79e7c93d911","Type":"ContainerStarted","Data":"49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb"}
Sep 17 18:22:10.710296 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.709896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" event={"ID":"cffed755-9673-47a1-9c9b-c79e7c93d911","Type":"ContainerStarted","Data":"3c31abe4e43ac79088d7f587da64752409439080e52cc9de3a869c98ce754a2c"}
Sep 17 18:22:10.711575 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.711550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" event={"ID":"da45b795-a647-476d-87bc-7d344102bbde","Type":"ContainerStarted","Data":"7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78"}
Sep 17 18:22:10.711703 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.711633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" event={"ID":"da45b795-a647-476d-87bc-7d344102bbde","Type":"ContainerStarted","Data":"bdc1e074787da40a1fc375d3c35966f3d949508c76c38cbdb21c19315e6b95ee"}
Sep 17 18:22:10.728758 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:10.728714    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" podStartSLOduration=1.7287001339999999 podStartE2EDuration="1.728700134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:10.726839173 +0000 UTC m=+2473.562608396" watchObservedRunningTime="2026-09-17 18:22:10.728700134 +0000 UTC m=+2473.564469357"
Sep 17 18:22:11.715409 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:11.715358    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:13.725122 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:13.725086    2565 generic.go:356] "Generic (PLEG): container finished" podID="da45b795-a647-476d-87bc-7d344102bbde" containerID="7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78" exitCode=0
Sep 17 18:22:13.725544 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:13.725159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" event={"ID":"da45b795-a647-476d-87bc-7d344102bbde","Type":"ContainerDied","Data":"7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78"}
Sep 17 18:22:14.731507 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:14.731476    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" event={"ID":"da45b795-a647-476d-87bc-7d344102bbde","Type":"ContainerStarted","Data":"79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5"}
Sep 17 18:22:14.753830 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:14.753762    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podStartSLOduration=3.188987645 podStartE2EDuration="5.753740801s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.564753156s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:14.751033067 +0000 UTC m=+2477.586802316" watchObservedRunningTime="2026-09-17 18:22:14.753740801 +0000 UTC m=+2477.589510025"
Sep 17 18:22:14.924282 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:14.924251    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:19.924741 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:19.924704    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:22:19.926237 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:19.926204    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:22:29.924897 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:29.924857    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:22:39.924694 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:39.924655    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:22:42.722332 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:42.722255    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:22:49.924808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:49.924770    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:22:59.924415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:22:59.924360    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:23:09.924536 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:09.924494    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:23:19.924762 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:19.924730    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:23:28.138137 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:28.138097    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"]
Sep 17 18:23:28.138571 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:28.138315    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy" containerID="cri-o://8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa" gracePeriod=30
Sep 17 18:23:29.924293 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:29.924256    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:23:33.388711 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.388668    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:23:33.444448 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444414    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqg7f\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-kube-api-access-zqg7f\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.444606 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-data\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.444606 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-socket\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.444606 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444532    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-certs\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.444606 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444559    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-istio-token\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.444606 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444605    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-credential-socket\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.444890 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444644    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-envoy\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.444890 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86967078-0345-4bf3-919a-8d874c0268f1-istiod-ca-cert\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.444890 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444727    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86967078-0345-4bf3-919a-8d874c0268f1-istio-podinfo\") pod \"86967078-0345-4bf3-919a-8d874c0268f1\" (UID: \"86967078-0345-4bf3-919a-8d874c0268f1\") "
Sep 17 18:23:33.445044 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.444999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:33.445044 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.445014    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:33.445627 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.445259    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:33.445627 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.445272    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:33.445828 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.445634    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86967078-0345-4bf3-919a-8d874c0268f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:33.447645 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.447617    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-kube-api-access-zqg7f" (OuterVolumeSpecName: "kube-api-access-zqg7f") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "kube-api-access-zqg7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:33.447838 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.447815    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:33.448053 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.448025    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:33.448166 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.448129    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86967078-0345-4bf3-919a-8d874c0268f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86967078-0345-4bf3-919a-8d874c0268f1" (UID: "86967078-0345-4bf3-919a-8d874c0268f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:23:33.545961 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.545927    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86967078-0345-4bf3-919a-8d874c0268f1-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:33.545961 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.545958    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86967078-0345-4bf3-919a-8d874c0268f1-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:33.545961 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.545966    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqg7f\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-kube-api-access-zqg7f\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:33.546199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.545975    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:33.546199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.545984    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:33.546199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.545992    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:33.546199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.546003    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86967078-0345-4bf3-919a-8d874c0268f1-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:33.546199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.546015    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:33.546199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:33.546025    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86967078-0345-4bf3-919a-8d874c0268f1-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:23:34.052170 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.052137    2565 generic.go:356] "Generic (PLEG): container finished" podID="86967078-0345-4bf3-919a-8d874c0268f1" containerID="8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa" exitCode=0
Sep 17 18:23:34.052319 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.052193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" event={"ID":"86967078-0345-4bf3-919a-8d874c0268f1","Type":"ContainerDied","Data":"8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa"}
Sep 17 18:23:34.052319 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.052205    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"
Sep 17 18:23:34.052319 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.052233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx" event={"ID":"86967078-0345-4bf3-919a-8d874c0268f1","Type":"ContainerDied","Data":"c86de2485e699ed71cdda0e7fd6b99169b4bbe8d8dcc119e0a00bdd785b9d3d3"}
Sep 17 18:23:34.052319 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.052255    2565 scope.go:117] "RemoveContainer" containerID="8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa"
Sep 17 18:23:34.060746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.060727    2565 scope.go:117] "RemoveContainer" containerID="247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75"
Sep 17 18:23:34.070218 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.070199    2565 scope.go:117] "RemoveContainer" containerID="8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa"
Sep 17 18:23:34.070556 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:23:34.070530    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa\": container with ID starting with 8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa not found: ID does not exist" containerID="8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa"
Sep 17 18:23:34.070631 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.070569    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa"} err="failed to get container status \"8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa\": rpc error: code = NotFound desc = could not find container \"8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa\": container with ID starting with 8ddaae00ac9cce9ab3f9e5673ca72b4eeb42d7a117af78771b39714fd813d8aa not found: ID does not exist"
Sep 17 18:23:34.070631 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.070596    2565 scope.go:117] "RemoveContainer" containerID="247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75"
Sep 17 18:23:34.070872 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:23:34.070852    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75\": container with ID starting with 247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75 not found: ID does not exist" containerID="247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75"
Sep 17 18:23:34.070917 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.070877    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75"} err="failed to get container status \"247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75\": rpc error: code = NotFound desc = could not find container \"247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75\": container with ID starting with 247b700c6e1d596631552ad9c0e912f1aff8747ee0fc1d463e93259b6fca1e75 not found: ID does not exist"
Sep 17 18:23:34.078343 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.078319    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"]
Sep 17 18:23:34.082301 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:34.082274    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-d4spx"]
Sep 17 18:23:35.349272 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:35.349238    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86967078-0345-4bf3-919a-8d874c0268f1" path="/var/lib/kubelet/pods/86967078-0345-4bf3-919a-8d874c0268f1/volumes"
Sep 17 18:23:39.924957 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:39.924919    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:23:49.924036 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:49.923998    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:23:59.923880 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:23:59.923843    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:24:09.934744 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:09.934714    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:24:09.942593 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:09.942569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:24:12.688063 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:12.687987    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"]
Sep 17 18:24:12.746763 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:12.746731    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"]
Sep 17 18:24:22.721758 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:22.721722    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:26.376299 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.376257    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"]
Sep 17 18:24:26.376793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.376688    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy"
Sep 17 18:24:26.376793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.376706    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy"
Sep 17 18:24:26.376793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.376721    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy"
Sep 17 18:24:26.376793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.376729    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy"
Sep 17 18:24:26.377030 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.376814    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy"
Sep 17 18:24:26.377030 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.376826    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86967078-0345-4bf3-919a-8d874c0268f1" containerName="istio-proxy"
Sep 17 18:24:26.380222 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.380202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.391694 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.391673    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"]
Sep 17 18:24:26.479500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.479463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggsp\" (UniqueName: \"kubernetes.io/projected/4f993993-adec-4f2e-a813-417cf4780398-kube-api-access-sggsp\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.479500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.479505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-tmp-dir\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.479701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.479526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-model-cache\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.479701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.479542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f993993-adec-4f2e-a813-417cf4780398-cabundle-cert\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.479701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.479603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f993993-adec-4f2e-a813-417cf4780398-tls-certs\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.479701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.479659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-home\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.479701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.479690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-dshm\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.479888 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.479730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-kserve-provision-location\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.580772 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.580735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggsp\" (UniqueName: \"kubernetes.io/projected/4f993993-adec-4f2e-a813-417cf4780398-kube-api-access-sggsp\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.580772 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.580773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-tmp-dir\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.580965 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.580792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-model-cache\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.580965 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.580827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f993993-adec-4f2e-a813-417cf4780398-cabundle-cert\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.580965 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.580853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f993993-adec-4f2e-a813-417cf4780398-tls-certs\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.580965 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.580904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-home\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.580965 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.580940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-dshm\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.581231 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.580974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-kserve-provision-location\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.581231 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.581185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-tmp-dir\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.581338 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.581305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-model-cache\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.581460 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.581333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-kserve-provision-location\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.581501 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.581474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-home\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.581566 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.581546    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f993993-adec-4f2e-a813-417cf4780398-cabundle-cert\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.583464 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.583435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-dshm\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.583629 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.583608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f993993-adec-4f2e-a813-417cf4780398-tls-certs\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.588334 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.588316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggsp\" (UniqueName: \"kubernetes.io/projected/4f993993-adec-4f2e-a813-417cf4780398-kube-api-access-sggsp\") pod \"stop-feature-test-kserve-75fff8659b-rs877\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.690248 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.690218    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:26.965773 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:26.965686    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"]
Sep 17 18:24:26.968120 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:24:26.968083    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f993993_adec_4f2e_a813_417cf4780398.slice/crio-f89b2c442cd846907deb4c3b1b505c235c13569ca423f2d3e49eda87eb6e8bde WatchSource:0}: Error finding container f89b2c442cd846907deb4c3b1b505c235c13569ca423f2d3e49eda87eb6e8bde: Status 404 returned error can't find the container with id f89b2c442cd846907deb4c3b1b505c235c13569ca423f2d3e49eda87eb6e8bde
Sep 17 18:24:27.269136 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:27.269046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" event={"ID":"4f993993-adec-4f2e-a813-417cf4780398","Type":"ContainerStarted","Data":"45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1"}
Sep 17 18:24:27.269136 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:27.269104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" event={"ID":"4f993993-adec-4f2e-a813-417cf4780398","Type":"ContainerStarted","Data":"f89b2c442cd846907deb4c3b1b505c235c13569ca423f2d3e49eda87eb6e8bde"}
Sep 17 18:24:27.712080 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:27.712030    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" containerID="cri-o://79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5" gracePeriod=45
Sep 17 18:24:27.766314 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:27.766267    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerName="main" containerID="cri-o://49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb" gracePeriod=45
Sep 17 18:24:27.936025 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:27.935987    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:24:28.465836 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.465803    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:24:28.497085 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497058    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-home\") pod \"da45b795-a647-476d-87bc-7d344102bbde\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") "
Sep 17 18:24:28.497238 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497105    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da45b795-a647-476d-87bc-7d344102bbde-cabundle-cert\") pod \"da45b795-a647-476d-87bc-7d344102bbde\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") "
Sep 17 18:24:28.497238 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-kserve-provision-location\") pod \"da45b795-a647-476d-87bc-7d344102bbde\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") "
Sep 17 18:24:28.497238 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497174    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-tmp-dir\") pod \"da45b795-a647-476d-87bc-7d344102bbde\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") "
Sep 17 18:24:28.497238 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497202    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da45b795-a647-476d-87bc-7d344102bbde-tls-certs\") pod \"da45b795-a647-476d-87bc-7d344102bbde\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") "
Sep 17 18:24:28.497505 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497259    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47fvl\" (UniqueName: \"kubernetes.io/projected/da45b795-a647-476d-87bc-7d344102bbde-kube-api-access-47fvl\") pod \"da45b795-a647-476d-87bc-7d344102bbde\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") "
Sep 17 18:24:28.497505 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497289    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-dshm\") pod \"da45b795-a647-476d-87bc-7d344102bbde\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") "
Sep 17 18:24:28.497505 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497310    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-model-cache\") pod \"da45b795-a647-476d-87bc-7d344102bbde\" (UID: \"da45b795-a647-476d-87bc-7d344102bbde\") "
Sep 17 18:24:28.498002 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497664    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da45b795-a647-476d-87bc-7d344102bbde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da45b795-a647-476d-87bc-7d344102bbde" (UID: "da45b795-a647-476d-87bc-7d344102bbde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:28.498002 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497694    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-model-cache" (OuterVolumeSpecName: "model-cache") pod "da45b795-a647-476d-87bc-7d344102bbde" (UID: "da45b795-a647-476d-87bc-7d344102bbde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:28.498002 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.497851    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-home" (OuterVolumeSpecName: "home") pod "da45b795-a647-476d-87bc-7d344102bbde" (UID: "da45b795-a647-476d-87bc-7d344102bbde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:28.500301 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.500261    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-dshm" (OuterVolumeSpecName: "dshm") pod "da45b795-a647-476d-87bc-7d344102bbde" (UID: "da45b795-a647-476d-87bc-7d344102bbde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:28.500459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.500367    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da45b795-a647-476d-87bc-7d344102bbde-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da45b795-a647-476d-87bc-7d344102bbde" (UID: "da45b795-a647-476d-87bc-7d344102bbde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:28.500910 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.500883    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da45b795-a647-476d-87bc-7d344102bbde-kube-api-access-47fvl" (OuterVolumeSpecName: "kube-api-access-47fvl") pod "da45b795-a647-476d-87bc-7d344102bbde" (UID: "da45b795-a647-476d-87bc-7d344102bbde"). InnerVolumeSpecName "kube-api-access-47fvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:28.515670 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.515647    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da45b795-a647-476d-87bc-7d344102bbde" (UID: "da45b795-a647-476d-87bc-7d344102bbde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:28.528350 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.528327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da45b795-a647-476d-87bc-7d344102bbde" (UID: "da45b795-a647-476d-87bc-7d344102bbde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:28.598345 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.598269    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.598345 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.598297    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da45b795-a647-476d-87bc-7d344102bbde-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.598345 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.598310    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.598345 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.598318    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.598345 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.598328    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da45b795-a647-476d-87bc-7d344102bbde-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.598345 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.598336    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fvl\" (UniqueName: \"kubernetes.io/projected/da45b795-a647-476d-87bc-7d344102bbde-kube-api-access-47fvl\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.598345 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.598343    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.598345 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:28.598351    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da45b795-a647-476d-87bc-7d344102bbde-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:29.277827 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.277792    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f993993-adec-4f2e-a813-417cf4780398" containerID="45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1" exitCode=0
Sep 17 18:24:29.278234 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.277864    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" event={"ID":"4f993993-adec-4f2e-a813-417cf4780398","Type":"ContainerDied","Data":"45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1"}
Sep 17 18:24:29.280057 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.280031    2565 generic.go:356] "Generic (PLEG): container finished" podID="da45b795-a647-476d-87bc-7d344102bbde" containerID="79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5" exitCode=0
Sep 17 18:24:29.280171 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.280113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" event={"ID":"da45b795-a647-476d-87bc-7d344102bbde","Type":"ContainerDied","Data":"79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5"}
Sep 17 18:24:29.280247 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.280192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf" event={"ID":"da45b795-a647-476d-87bc-7d344102bbde","Type":"ContainerDied","Data":"bdc1e074787da40a1fc375d3c35966f3d949508c76c38cbdb21c19315e6b95ee"}
Sep 17 18:24:29.280247 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.280243    2565 scope.go:117] "RemoveContainer" containerID="79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5"
Sep 17 18:24:29.280352 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.280261    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"
Sep 17 18:24:29.295158 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.295142    2565 scope.go:117] "RemoveContainer" containerID="7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78"
Sep 17 18:24:29.309187 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.309171    2565 scope.go:117] "RemoveContainer" containerID="79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5"
Sep 17 18:24:29.309522 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:24:29.309503    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5\": container with ID starting with 79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5 not found: ID does not exist" containerID="79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5"
Sep 17 18:24:29.309595 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.309530    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5"} err="failed to get container status \"79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5\": rpc error: code = NotFound desc = could not find container \"79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5\": container with ID starting with 79b743a750c67805f7af50905c6717b0028ac326f04c8019ac456946d67dc9d5 not found: ID does not exist"
Sep 17 18:24:29.309595 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.309547    2565 scope.go:117] "RemoveContainer" containerID="7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78"
Sep 17 18:24:29.309793 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:24:29.309776    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78\": container with ID starting with 7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78 not found: ID does not exist" containerID="7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78"
Sep 17 18:24:29.309837 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.309799    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78"} err="failed to get container status \"7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78\": rpc error: code = NotFound desc = could not find container \"7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78\": container with ID starting with 7a536c7261fab38a58c44388369c40cb190572cc61c718d014d1c821e8ac2f78 not found: ID does not exist"
Sep 17 18:24:29.310128 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.310107    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"]
Sep 17 18:24:29.313315 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.313296    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-x2kkf"]
Sep 17 18:24:29.347897 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:29.347865    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da45b795-a647-476d-87bc-7d344102bbde" path="/var/lib/kubelet/pods/da45b795-a647-476d-87bc-7d344102bbde/volumes"
Sep 17 18:24:30.285477 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:30.285437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" event={"ID":"4f993993-adec-4f2e-a813-417cf4780398","Type":"ContainerStarted","Data":"655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3"}
Sep 17 18:24:30.306414 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:30.306352    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podStartSLOduration=2.164661355 podStartE2EDuration="4.306337039s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.141675684s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:30.304764451 +0000 UTC m=+2613.140533676" watchObservedRunningTime="2026-09-17 18:24:30.306337039 +0000 UTC m=+2613.142106261"
Sep 17 18:24:30.690804 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:30.690772    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:32.722537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:32.722500    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:36.691412 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:36.691360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:24:36.692794 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:36.692768    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:24:42.721707 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:42.721666    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:42.722172 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:42.721776    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:24:46.690852 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:46.690803    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:24:52.722343 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:52.722295    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:56.691569 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:56.691525    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:24:58.007675 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.007651    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:24:58.151102 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.151021    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db5ww\" (UniqueName: \"kubernetes.io/projected/cffed755-9673-47a1-9c9b-c79e7c93d911-kube-api-access-db5ww\") pod \"cffed755-9673-47a1-9c9b-c79e7c93d911\" (UID: \"cffed755-9673-47a1-9c9b-c79e7c93d911\") "
Sep 17 18:24:58.151102 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.151065    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cffed755-9673-47a1-9c9b-c79e7c93d911-tls-certs\") pod \"cffed755-9673-47a1-9c9b-c79e7c93d911\" (UID: \"cffed755-9673-47a1-9c9b-c79e7c93d911\") "
Sep 17 18:24:58.153166 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.153138    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffed755-9673-47a1-9c9b-c79e7c93d911-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cffed755-9673-47a1-9c9b-c79e7c93d911" (UID: "cffed755-9673-47a1-9c9b-c79e7c93d911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:58.153290 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.153187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffed755-9673-47a1-9c9b-c79e7c93d911-kube-api-access-db5ww" (OuterVolumeSpecName: "kube-api-access-db5ww") pod "cffed755-9673-47a1-9c9b-c79e7c93d911" (UID: "cffed755-9673-47a1-9c9b-c79e7c93d911"). InnerVolumeSpecName "kube-api-access-db5ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:58.252628 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.252596    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db5ww\" (UniqueName: \"kubernetes.io/projected/cffed755-9673-47a1-9c9b-c79e7c93d911-kube-api-access-db5ww\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:58.252628 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.252621    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cffed755-9673-47a1-9c9b-c79e7c93d911-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:24:58.396147 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.396107    2565 generic.go:356] "Generic (PLEG): container finished" podID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerID="49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb" exitCode=0
Sep 17 18:24:58.396147 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.396143    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"
Sep 17 18:24:58.396433 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.396189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" event={"ID":"cffed755-9673-47a1-9c9b-c79e7c93d911","Type":"ContainerDied","Data":"49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb"}
Sep 17 18:24:58.396433 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.396243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7" event={"ID":"cffed755-9673-47a1-9c9b-c79e7c93d911","Type":"ContainerDied","Data":"3c31abe4e43ac79088d7f587da64752409439080e52cc9de3a869c98ce754a2c"}
Sep 17 18:24:58.396433 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.396272    2565 scope.go:117] "RemoveContainer" containerID="49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb"
Sep 17 18:24:58.404736 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.404718    2565 scope.go:117] "RemoveContainer" containerID="49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb"
Sep 17 18:24:58.404989 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:24:58.404970    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb\": container with ID starting with 49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb not found: ID does not exist" containerID="49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb"
Sep 17 18:24:58.405078 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.405002    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb"} err="failed to get container status \"49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb\": rpc error: code = NotFound desc = could not find container \"49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb\": container with ID starting with 49ef5a5a76566e275b4a4b40f3a801900dc37beb315505c612fdb5701aa9a9eb not found: ID does not exist"
Sep 17 18:24:58.416585 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.416534    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"]
Sep 17 18:24:58.420239 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:58.420217    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gzfx7"]
Sep 17 18:24:59.350963 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:24:59.350931    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" path="/var/lib/kubelet/pods/cffed755-9673-47a1-9c9b-c79e7c93d911/volumes"
Sep 17 18:25:06.691090 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:25:06.691048    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:25:16.691688 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:25:16.691630    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:25:26.690826 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:25:26.690781    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:25:36.691133 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:25:36.691092    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:25:46.691406 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:25:46.691301    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:25:56.691137 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:25:56.691095    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:25:59.445331 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:25:59.445300    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:25:59.449916 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:25:59.449891    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:26:06.690958 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:06.690913    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:26:16.690652 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:16.690607    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:26:26.701207 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:26.701167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:26:26.709028 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:26.709006    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:26:28.035232 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:28.035189    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"]
Sep 17 18:26:43.063014 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.062948    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" containerID="cri-o://655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3" gracePeriod=45
Sep 17 18:26:43.702295 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.702245    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 18:26:43.801077 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.801054    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:26:43.808226 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.808199    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f993993-adec-4f2e-a813-417cf4780398" containerID="655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3" exitCode=0
Sep 17 18:26:43.808341 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.808237    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"
Sep 17 18:26:43.808341 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.808272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" event={"ID":"4f993993-adec-4f2e-a813-417cf4780398","Type":"ContainerDied","Data":"655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3"}
Sep 17 18:26:43.808341 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.808321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877" event={"ID":"4f993993-adec-4f2e-a813-417cf4780398","Type":"ContainerDied","Data":"f89b2c442cd846907deb4c3b1b505c235c13569ca423f2d3e49eda87eb6e8bde"}
Sep 17 18:26:43.808493 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.808342    2565 scope.go:117] "RemoveContainer" containerID="655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3"
Sep 17 18:26:43.815862 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.815847    2565 scope.go:117] "RemoveContainer" containerID="45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1"
Sep 17 18:26:43.823111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.823096    2565 scope.go:117] "RemoveContainer" containerID="655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3"
Sep 17 18:26:43.823385 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:26:43.823362    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3\": container with ID starting with 655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3 not found: ID does not exist" containerID="655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3"
Sep 17 18:26:43.823535 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.823404    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3"} err="failed to get container status \"655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3\": rpc error: code = NotFound desc = could not find container \"655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3\": container with ID starting with 655a8373c868ac179373b55963bb36a5d2fd21c06ee06fcc513fc78e14a0d5d3 not found: ID does not exist"
Sep 17 18:26:43.823535 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.823424    2565 scope.go:117] "RemoveContainer" containerID="45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1"
Sep 17 18:26:43.823740 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:26:43.823697    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1\": container with ID starting with 45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1 not found: ID does not exist" containerID="45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1"
Sep 17 18:26:43.823788 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.823750    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1"} err="failed to get container status \"45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1\": rpc error: code = NotFound desc = could not find container \"45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1\": container with ID starting with 45c680e591dd2886701f92fdf5b62088dc940cbbee7a3f6e9b79413e7af4e3d1 not found: ID does not exist"
Sep 17 18:26:43.881701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.881672    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-tmp-dir\") pod \"4f993993-adec-4f2e-a813-417cf4780398\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") "
Sep 17 18:26:43.881835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.881719    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f993993-adec-4f2e-a813-417cf4780398-cabundle-cert\") pod \"4f993993-adec-4f2e-a813-417cf4780398\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") "
Sep 17 18:26:43.881835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.881746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-kserve-provision-location\") pod \"4f993993-adec-4f2e-a813-417cf4780398\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") "
Sep 17 18:26:43.881835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.881780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f993993-adec-4f2e-a813-417cf4780398-tls-certs\") pod \"4f993993-adec-4f2e-a813-417cf4780398\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") "
Sep 17 18:26:43.881835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.881807    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-model-cache\") pod \"4f993993-adec-4f2e-a813-417cf4780398\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") "
Sep 17 18:26:43.882056 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.881840    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-dshm\") pod \"4f993993-adec-4f2e-a813-417cf4780398\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") "
Sep 17 18:26:43.882056 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.881884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-home\") pod \"4f993993-adec-4f2e-a813-417cf4780398\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") "
Sep 17 18:26:43.882056 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.881919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sggsp\" (UniqueName: \"kubernetes.io/projected/4f993993-adec-4f2e-a813-417cf4780398-kube-api-access-sggsp\") pod \"4f993993-adec-4f2e-a813-417cf4780398\" (UID: \"4f993993-adec-4f2e-a813-417cf4780398\") "
Sep 17 18:26:43.882200 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.882124    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f993993-adec-4f2e-a813-417cf4780398" (UID: "4f993993-adec-4f2e-a813-417cf4780398"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:43.882614 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.882588    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f993993-adec-4f2e-a813-417cf4780398-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f993993-adec-4f2e-a813-417cf4780398" (UID: "4f993993-adec-4f2e-a813-417cf4780398"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:43.882723 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.882635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-home" (OuterVolumeSpecName: "home") pod "4f993993-adec-4f2e-a813-417cf4780398" (UID: "4f993993-adec-4f2e-a813-417cf4780398"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:43.884047 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.884023    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-dshm" (OuterVolumeSpecName: "dshm") pod "4f993993-adec-4f2e-a813-417cf4780398" (UID: "4f993993-adec-4f2e-a813-417cf4780398"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:43.884425 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.884383    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f993993-adec-4f2e-a813-417cf4780398-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f993993-adec-4f2e-a813-417cf4780398" (UID: "4f993993-adec-4f2e-a813-417cf4780398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:43.884498 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.884415    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f993993-adec-4f2e-a813-417cf4780398-kube-api-access-sggsp" (OuterVolumeSpecName: "kube-api-access-sggsp") pod "4f993993-adec-4f2e-a813-417cf4780398" (UID: "4f993993-adec-4f2e-a813-417cf4780398"). InnerVolumeSpecName "kube-api-access-sggsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:43.899616 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.899586    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f993993-adec-4f2e-a813-417cf4780398" (UID: "4f993993-adec-4f2e-a813-417cf4780398"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:43.912148 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.912122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f993993-adec-4f2e-a813-417cf4780398" (UID: "4f993993-adec-4f2e-a813-417cf4780398"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:43.982673 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.982650    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:26:43.982673 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.982671    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f993993-adec-4f2e-a813-417cf4780398-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:26:43.982790 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.982681    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:26:43.982790 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.982691    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f993993-adec-4f2e-a813-417cf4780398-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:26:43.982790 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.982699    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:26:43.982790 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.982707    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:26:43.982790 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.982714    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f993993-adec-4f2e-a813-417cf4780398-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:26:43.982790 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:43.982722    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sggsp\" (UniqueName: \"kubernetes.io/projected/4f993993-adec-4f2e-a813-417cf4780398-kube-api-access-sggsp\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:26:44.128483 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:44.128454    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"]
Sep 17 18:26:44.132337 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:44.132312    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-75fff8659b-rs877"]
Sep 17 18:26:45.349139 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:26:45.349096    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f993993-adec-4f2e-a813-417cf4780398" path="/var/lib/kubelet/pods/4f993993-adec-4f2e-a813-417cf4780398/volumes"
Sep 17 18:27:28.163661 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163628    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"]
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163892    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="storage-initializer"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163904    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="storage-initializer"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163918    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163923    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163933    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="storage-initializer"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163939    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="storage-initializer"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163956    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerName="main"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163963    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerName="main"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163970    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.163975    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.164024    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da45b795-a647-476d-87bc-7d344102bbde" containerName="main"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.164035    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f993993-adec-4f2e-a813-417cf4780398" containerName="main"
Sep 17 18:27:28.164055 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.164041    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffed755-9673-47a1-9c9b-c79e7c93d911" containerName="main"
Sep 17 18:27:28.168038 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.168017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.170233 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.170208    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 18:27:28.170352 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.170268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 18:27:28.170352 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.170279    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-l2978\""
Sep 17 18:27:28.170484 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.170368    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:27:28.170484 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.170449    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 18:27:28.170629 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.170604    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 18:27:28.180459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.180437    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"]
Sep 17 18:27:28.220432 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.220390    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdmt\" (UniqueName: \"kubernetes.io/projected/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kube-api-access-zjdmt\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.220566 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.220439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.220566 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.220507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.220566 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.220544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.220680 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.220581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.220680 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.220611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.220680 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.220632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.220680 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.220660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321184 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321184 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdmt\" (UniqueName: \"kubernetes.io/projected/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kube-api-access-zjdmt\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321727 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321673    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321787 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321717    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321787 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.321902 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.321837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.322088 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.322069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.323620 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.323604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.323948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.323930    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.329155 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.329132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdmt\" (UniqueName: \"kubernetes.io/projected/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kube-api-access-zjdmt\") pod \"custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.437923 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.437848    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"]
Sep 17 18:27:28.440924 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.440903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:28.442965 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.442939    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-clp44\""
Sep 17 18:27:28.449818 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.449793    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"]
Sep 17 18:27:28.477909 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.477885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:28.522720 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.522685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pvlq\" (UniqueName: \"kubernetes.io/projected/987e9fc2-50c8-488e-9abc-c38cc6228d1a-kube-api-access-8pvlq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s\" (UID: \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:28.522921 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.522750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987e9fc2-50c8-488e-9abc-c38cc6228d1a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s\" (UID: \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:28.623620 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.623587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987e9fc2-50c8-488e-9abc-c38cc6228d1a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s\" (UID: \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:28.623776 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.623650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pvlq\" (UniqueName: \"kubernetes.io/projected/987e9fc2-50c8-488e-9abc-c38cc6228d1a-kube-api-access-8pvlq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s\" (UID: \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:28.626085 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.626045    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987e9fc2-50c8-488e-9abc-c38cc6228d1a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s\" (UID: \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:28.631549 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.631523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pvlq\" (UniqueName: \"kubernetes.io/projected/987e9fc2-50c8-488e-9abc-c38cc6228d1a-kube-api-access-8pvlq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s\" (UID: \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:28.752176 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.752090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:28.760165 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.760138    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"]
Sep 17 18:27:28.764224 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:27:28.764198    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3537cfe_57e3_4bc3_ba7f_f29954fe0829.slice/crio-e4c91cb822c6e0216990382e5e64ec95bb09151e6e1350c66287dda8ada5d3a5 WatchSource:0}: Error finding container e4c91cb822c6e0216990382e5e64ec95bb09151e6e1350c66287dda8ada5d3a5: Status 404 returned error can't find the container with id e4c91cb822c6e0216990382e5e64ec95bb09151e6e1350c66287dda8ada5d3a5
Sep 17 18:27:28.974642 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.974606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" event={"ID":"c3537cfe-57e3-4bc3-ba7f-f29954fe0829","Type":"ContainerStarted","Data":"dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a"}
Sep 17 18:27:28.974817 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:28.974669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" event={"ID":"c3537cfe-57e3-4bc3-ba7f-f29954fe0829","Type":"ContainerStarted","Data":"e4c91cb822c6e0216990382e5e64ec95bb09151e6e1350c66287dda8ada5d3a5"}
Sep 17 18:27:29.028452 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:29.028419    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"]
Sep 17 18:27:29.030690 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:27:29.030663    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987e9fc2_50c8_488e_9abc_c38cc6228d1a.slice/crio-8112f2a736fb4502188fc47cdb1554f275aea27c7c91e0e27e9c56942f112369 WatchSource:0}: Error finding container 8112f2a736fb4502188fc47cdb1554f275aea27c7c91e0e27e9c56942f112369: Status 404 returned error can't find the container with id 8112f2a736fb4502188fc47cdb1554f275aea27c7c91e0e27e9c56942f112369
Sep 17 18:27:29.982084 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:29.982050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" event={"ID":"987e9fc2-50c8-488e-9abc-c38cc6228d1a","Type":"ContainerStarted","Data":"b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205"}
Sep 17 18:27:29.982669 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:29.982640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" event={"ID":"987e9fc2-50c8-488e-9abc-c38cc6228d1a","Type":"ContainerStarted","Data":"8112f2a736fb4502188fc47cdb1554f275aea27c7c91e0e27e9c56942f112369"}
Sep 17 18:27:29.997559 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:29.997511    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" podStartSLOduration=1.9974930180000001 podStartE2EDuration="1.997493018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:29.996546338 +0000 UTC m=+2792.832315560" watchObservedRunningTime="2026-09-17 18:27:29.997493018 +0000 UTC m=+2792.833262254"
Sep 17 18:27:30.994301 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:30.994237    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:27:31.999372 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:31.999336    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerID="dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a" exitCode=0
Sep 17 18:27:31.999877 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:31.999858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" event={"ID":"c3537cfe-57e3-4bc3-ba7f-f29954fe0829","Type":"ContainerDied","Data":"dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a"}
Sep 17 18:27:33.004836 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:33.004802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" event={"ID":"c3537cfe-57e3-4bc3-ba7f-f29954fe0829","Type":"ContainerStarted","Data":"4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93"}
Sep 17 18:27:33.028717 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:33.028656    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podStartSLOduration=2.568702917 podStartE2EDuration="5.028636931s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.459934014s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:33.027932741 +0000 UTC m=+2795.863702000" watchObservedRunningTime="2026-09-17 18:27:33.028636931 +0000 UTC m=+2795.864406153"
Sep 17 18:27:38.479023 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:38.478980    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:38.479449 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:38.479051    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:27:38.480424 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:38.480371    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:27:48.478379 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:48.478337    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:27:58.478507 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:27:58.478452    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:28:02.001808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:28:02.001771    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:28:08.478825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:28:08.478780    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:28:18.478683 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:28:18.478638    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:28:28.478580 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:28:28.478536    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:28:38.478672 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:28:38.478620    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:28:48.479372 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:28:48.479330    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:28:58.478877 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:28:58.478831    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:29:08.478856 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:08.478802    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:29:18.478905 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:18.478850    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:29:28.488782 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:28.488742    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:29:28.496825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:28.496793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:29:39.659789 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:39.659752    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"]
Sep 17 18:29:39.664010 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:39.663977    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"]
Sep 17 18:29:42.001630 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:42.001586    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:52.001499 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:52.001454    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:54.683825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:54.683752    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main" containerID="cri-o://4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93" gracePeriod=45
Sep 17 18:29:54.684294 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:54.683853    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main" containerID="cri-o://b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205" gracePeriod=45
Sep 17 18:29:55.428297 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.428266    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:29:55.525799 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.525717    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-dshm\") pod \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") "
Sep 17 18:29:55.525799 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.525757    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kserve-provision-location\") pod \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") "
Sep 17 18:29:55.525799 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.525785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tmp-dir\") pod \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") "
Sep 17 18:29:55.526095 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.525823    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-model-cache\") pod \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") "
Sep 17 18:29:55.526095 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.525843    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjdmt\" (UniqueName: \"kubernetes.io/projected/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kube-api-access-zjdmt\") pod \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") "
Sep 17 18:29:55.526095 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.525873    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-cabundle-cert\") pod \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") "
Sep 17 18:29:55.526095 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.525912    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tls-certs\") pod \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") "
Sep 17 18:29:55.526095 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.525939    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-home\") pod \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\" (UID: \"c3537cfe-57e3-4bc3-ba7f-f29954fe0829\") "
Sep 17 18:29:55.526352 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.526116    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3537cfe-57e3-4bc3-ba7f-f29954fe0829" (UID: "c3537cfe-57e3-4bc3-ba7f-f29954fe0829"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.526426 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.526385    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3537cfe-57e3-4bc3-ba7f-f29954fe0829" (UID: "c3537cfe-57e3-4bc3-ba7f-f29954fe0829"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:55.526830 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.526803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-home" (OuterVolumeSpecName: "home") pod "c3537cfe-57e3-4bc3-ba7f-f29954fe0829" (UID: "c3537cfe-57e3-4bc3-ba7f-f29954fe0829"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.528257 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.528221    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kube-api-access-zjdmt" (OuterVolumeSpecName: "kube-api-access-zjdmt") pod "c3537cfe-57e3-4bc3-ba7f-f29954fe0829" (UID: "c3537cfe-57e3-4bc3-ba7f-f29954fe0829"). InnerVolumeSpecName "kube-api-access-zjdmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:55.528257 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.528238    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-dshm" (OuterVolumeSpecName: "dshm") pod "c3537cfe-57e3-4bc3-ba7f-f29954fe0829" (UID: "c3537cfe-57e3-4bc3-ba7f-f29954fe0829"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.528504 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.528277    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3537cfe-57e3-4bc3-ba7f-f29954fe0829" (UID: "c3537cfe-57e3-4bc3-ba7f-f29954fe0829"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:55.539124 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.539097    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3537cfe-57e3-4bc3-ba7f-f29954fe0829" (UID: "c3537cfe-57e3-4bc3-ba7f-f29954fe0829"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.549151 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.549123    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerID="4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93" exitCode=0
Sep 17 18:29:55.549272 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.549169    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"
Sep 17 18:29:55.549272 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.549201    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" event={"ID":"c3537cfe-57e3-4bc3-ba7f-f29954fe0829","Type":"ContainerDied","Data":"4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93"}
Sep 17 18:29:55.549272 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.549246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45" event={"ID":"c3537cfe-57e3-4bc3-ba7f-f29954fe0829","Type":"ContainerDied","Data":"e4c91cb822c6e0216990382e5e64ec95bb09151e6e1350c66287dda8ada5d3a5"}
Sep 17 18:29:55.549272 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.549270    2565 scope.go:117] "RemoveContainer" containerID="4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93"
Sep 17 18:29:55.550655 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.550631    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3537cfe-57e3-4bc3-ba7f-f29954fe0829" (UID: "c3537cfe-57e3-4bc3-ba7f-f29954fe0829"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.557355 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.557327    2565 scope.go:117] "RemoveContainer" containerID="dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a"
Sep 17 18:29:55.564446 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.564427    2565 scope.go:117] "RemoveContainer" containerID="4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93"
Sep 17 18:29:55.564718 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:29:55.564700    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93\": container with ID starting with 4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93 not found: ID does not exist" containerID="4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93"
Sep 17 18:29:55.564780 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.564730    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93"} err="failed to get container status \"4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93\": rpc error: code = NotFound desc = could not find container \"4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93\": container with ID starting with 4cbf276923d12f2e10fd91393647fd4e745dd0d0c1efa544c51b342c0e291e93 not found: ID does not exist"
Sep 17 18:29:55.564780 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.564748    2565 scope.go:117] "RemoveContainer" containerID="dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a"
Sep 17 18:29:55.565001 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:29:55.564981    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a\": container with ID starting with dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a not found: ID does not exist" containerID="dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a"
Sep 17 18:29:55.565059 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.565010    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a"} err="failed to get container status \"dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a\": rpc error: code = NotFound desc = could not find container \"dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a\": container with ID starting with dfb2c218530f680a6e490afac7c793ee8b9b0372fc8f748861224444e607073a not found: ID does not exist"
Sep 17 18:29:55.626742 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.626713    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.626742 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.626740    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.626908 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.626751    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.626908 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.626763    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.626908 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.626775    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.626908 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.626786    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.626908 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.626800    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.626908 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.626811    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjdmt\" (UniqueName: \"kubernetes.io/projected/c3537cfe-57e3-4bc3-ba7f-f29954fe0829-kube-api-access-zjdmt\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.878768 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.878737    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"]
Sep 17 18:29:55.891141 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:55.891107    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-b98ccb957-jvp45"]
Sep 17 18:29:57.348231 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:29:57.348199    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" path="/var/lib/kubelet/pods/c3537cfe-57e3-4bc3-ba7f-f29954fe0829/volumes"
Sep 17 18:30:02.002052 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:02.002011    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:02.002507 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:02.002121    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:30:12.001373 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:12.001277    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:22.001513 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:22.001471    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:24.923652 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:24.923627    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:30:25.058646 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.058559    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987e9fc2-50c8-488e-9abc-c38cc6228d1a-tls-certs\") pod \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\" (UID: \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\") "
Sep 17 18:30:25.058646 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.058628    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pvlq\" (UniqueName: \"kubernetes.io/projected/987e9fc2-50c8-488e-9abc-c38cc6228d1a-kube-api-access-8pvlq\") pod \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\" (UID: \"987e9fc2-50c8-488e-9abc-c38cc6228d1a\") "
Sep 17 18:30:25.060702 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.060672    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987e9fc2-50c8-488e-9abc-c38cc6228d1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "987e9fc2-50c8-488e-9abc-c38cc6228d1a" (UID: "987e9fc2-50c8-488e-9abc-c38cc6228d1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:25.060853 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.060756    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987e9fc2-50c8-488e-9abc-c38cc6228d1a-kube-api-access-8pvlq" (OuterVolumeSpecName: "kube-api-access-8pvlq") pod "987e9fc2-50c8-488e-9abc-c38cc6228d1a" (UID: "987e9fc2-50c8-488e-9abc-c38cc6228d1a"). InnerVolumeSpecName "kube-api-access-8pvlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:25.159186 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.159158    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987e9fc2-50c8-488e-9abc-c38cc6228d1a-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:30:25.159186 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.159181    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pvlq\" (UniqueName: \"kubernetes.io/projected/987e9fc2-50c8-488e-9abc-c38cc6228d1a-kube-api-access-8pvlq\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:30:25.664119 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.664089    2565 generic.go:356] "Generic (PLEG): container finished" podID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerID="b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205" exitCode=0
Sep 17 18:30:25.664275 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.664149    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"
Sep 17 18:30:25.664275 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.664182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" event={"ID":"987e9fc2-50c8-488e-9abc-c38cc6228d1a","Type":"ContainerDied","Data":"b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205"}
Sep 17 18:30:25.664275 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.664240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s" event={"ID":"987e9fc2-50c8-488e-9abc-c38cc6228d1a","Type":"ContainerDied","Data":"8112f2a736fb4502188fc47cdb1554f275aea27c7c91e0e27e9c56942f112369"}
Sep 17 18:30:25.664275 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.664265    2565 scope.go:117] "RemoveContainer" containerID="b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205"
Sep 17 18:30:25.672469 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.672448    2565 scope.go:117] "RemoveContainer" containerID="b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205"
Sep 17 18:30:25.672714 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:30:25.672697    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205\": container with ID starting with b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205 not found: ID does not exist" containerID="b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205"
Sep 17 18:30:25.672767 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.672722    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205"} err="failed to get container status \"b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205\": rpc error: code = NotFound desc = could not find container \"b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205\": container with ID starting with b0e6965df24dd79e15b29f9722b5a928dc460a73509fe6116a16571b173a9205 not found: ID does not exist"
Sep 17 18:30:25.679249 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.679228    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"]
Sep 17 18:30:25.683058 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:25.683037    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48cn54s"]
Sep 17 18:30:27.348067 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:27.348032    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" path="/var/lib/kubelet/pods/987e9fc2-50c8-488e-9abc-c38cc6228d1a/volumes"
Sep 17 18:30:30.098728 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.098698    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"]
Sep 17 18:30:30.099112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.098944    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main"
Sep 17 18:30:30.099112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.098954    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main"
Sep 17 18:30:30.099112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.098969    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="storage-initializer"
Sep 17 18:30:30.099112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.098974    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="storage-initializer"
Sep 17 18:30:30.099112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.098986    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main"
Sep 17 18:30:30.099112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.098991    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main"
Sep 17 18:30:30.099112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.099034    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="987e9fc2-50c8-488e-9abc-c38cc6228d1a" containerName="main"
Sep 17 18:30:30.099112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.099044    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3537cfe-57e3-4bc3-ba7f-f29954fe0829" containerName="main"
Sep 17 18:30:30.101633 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.101614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.104360 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.104337    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:30:30.104500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.104383    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:30:30.104500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.104409    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:30:30.104500 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.104471    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-c5mlg\""
Sep 17 18:30:30.116097 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.115161    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"]
Sep 17 18:30:30.195332 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.195332 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxxfr\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-kube-api-access-jxxfr\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.195557 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.195557 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.195557 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.195557 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.195557 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.195557 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.195751 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.195600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296352 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296352 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296604 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296604 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296604 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296771 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296771 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxxfr\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-kube-api-access-jxxfr\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296771 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296690    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296771 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296972 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296972 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.296972 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.296865    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.297083 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.297001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.297333 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.297314    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.298939 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.298915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.299177 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.299159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.306274 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.306250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.306498 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.306472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxxfr\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-kube-api-access-jxxfr\") pod \"router-gateway-2-openshift-default-844b774c6-gkngz\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.414728 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.414702    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:30.701275 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.701200    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"]
Sep 17 18:30:30.704360 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:30:30.704333    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c9e448_fe8b_4d1d_9861_3fefb9b920cb.slice/crio-a9d429a737a7cb9cc8fca241799e65a27f50d39bcd4867411bd3361588d70471 WatchSource:0}: Error finding container a9d429a737a7cb9cc8fca241799e65a27f50d39bcd4867411bd3361588d70471: Status 404 returned error can't find the container with id a9d429a737a7cb9cc8fca241799e65a27f50d39bcd4867411bd3361588d70471
Sep 17 18:30:30.706338 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.706305    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:30:30.706426 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.706371    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:30:30.706426 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:30.706419    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:30:31.688743 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:31.688706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" event={"ID":"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb","Type":"ContainerStarted","Data":"e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05"}
Sep 17 18:30:31.689257 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:31.688758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" event={"ID":"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb","Type":"ContainerStarted","Data":"a9d429a737a7cb9cc8fca241799e65a27f50d39bcd4867411bd3361588d70471"}
Sep 17 18:30:31.708978 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:31.708926    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" podStartSLOduration=1.70891229 podStartE2EDuration="1.70891229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:31.707724864 +0000 UTC m=+2974.543494087" watchObservedRunningTime="2026-09-17 18:30:31.70891229 +0000 UTC m=+2974.544681512"
Sep 17 18:30:32.415285 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:32.415249    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:32.420482 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:32.420455    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:32.692493 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:32.692383    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:32.693285 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:32.693261    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:42.728805 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:42.728725    2565 generic.go:356] "Generic (PLEG): container finished" podID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerID="e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05" exitCode=0
Sep 17 18:30:42.728805 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:42.728776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" event={"ID":"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb","Type":"ContainerDied","Data":"e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05"}
Sep 17 18:30:42.729302 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:42.729089    2565 scope.go:117] "RemoveContainer" containerID="e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05"
Sep 17 18:30:42.730005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:42.729977    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:30:42.730071 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:42.730045    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:30:42.730110 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:42.730076    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:30:43.733906 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:43.733870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" event={"ID":"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb","Type":"ContainerStarted","Data":"e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf"}
Sep 17 18:30:44.415625 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:44.415587    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:44.416584 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:44.416557    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 18:30:45.415543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:45.415500    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 18:30:46.415883 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:46.415836    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 18:30:47.416040 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:47.416001    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 18:30:47.692893 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:47.692802    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:48.415506 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:48.415462    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 18:30:49.418936 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:49.418905    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:49.419969 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:49.419950    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:30:59.468334 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:59.468309    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:30:59.473414 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:30:59.473377    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:31:20.641924 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.641892    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"]
Sep 17 18:31:20.644103 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.644086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.646282 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.646258    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-g9lxl\""
Sep 17 18:31:20.646564 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.646542    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:31:20.646695 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.646566    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-qdtrb\""
Sep 17 18:31:20.646695 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.646592    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:31:20.646695 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.646566    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 18:31:20.662740 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.662718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"]
Sep 17 18:31:20.800741 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.800696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f58ae4-1118-4c12-8630-88951ec7db9e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.800741 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.800743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfsmt\" (UniqueName: \"kubernetes.io/projected/05f58ae4-1118-4c12-8630-88951ec7db9e-kube-api-access-vfsmt\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.800998 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.800821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-model-cache\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.800998 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.800905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-dshm\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.800998 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.800938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-home\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.801164 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.801002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f58ae4-1118-4c12-8630-88951ec7db9e-tls-certs\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.801164 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.801039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.801164 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.801093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902443 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-dshm\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902443 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-home\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902443 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f58ae4-1118-4c12-8630-88951ec7db9e-tls-certs\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902725 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902725 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902725 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f58ae4-1118-4c12-8630-88951ec7db9e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902725 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfsmt\" (UniqueName: \"kubernetes.io/projected/05f58ae4-1118-4c12-8630-88951ec7db9e-kube-api-access-vfsmt\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902725 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-model-cache\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.902989 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.902839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-home\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.903153 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.903123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.903243 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.903200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.903325 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.903308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-model-cache\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.903381 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.903333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f58ae4-1118-4c12-8630-88951ec7db9e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.904722 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.904703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-dshm\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.904857 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.904842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f58ae4-1118-4c12-8630-88951ec7db9e-tls-certs\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.910184 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.910159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfsmt\" (UniqueName: \"kubernetes.io/projected/05f58ae4-1118-4c12-8630-88951ec7db9e-kube-api-access-vfsmt\") pod \"router-with-refs-pd-test-kserve-944bd467c-hhhg8\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:20.954198 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:20.954175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:21.231461 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:21.231426    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"]
Sep 17 18:31:21.234305 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:31:21.234280    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f58ae4_1118_4c12_8630_88951ec7db9e.slice/crio-e20aa836812b96afb4d884e38a91cefb4ac2d13ac8086b55f3f39021fa5fb826 WatchSource:0}: Error finding container e20aa836812b96afb4d884e38a91cefb4ac2d13ac8086b55f3f39021fa5fb826: Status 404 returned error can't find the container with id e20aa836812b96afb4d884e38a91cefb4ac2d13ac8086b55f3f39021fa5fb826
Sep 17 18:31:21.236260 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:21.236241    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:31:21.878753 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:21.878714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" event={"ID":"05f58ae4-1118-4c12-8630-88951ec7db9e","Type":"ContainerStarted","Data":"e20aa836812b96afb4d884e38a91cefb4ac2d13ac8086b55f3f39021fa5fb826"}
Sep 17 18:31:23.887353 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:23.887311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" event={"ID":"05f58ae4-1118-4c12-8630-88951ec7db9e","Type":"ContainerStarted","Data":"d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74"}
Sep 17 18:31:23.887834 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:23.887459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:24.892800 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:24.892755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" event={"ID":"05f58ae4-1118-4c12-8630-88951ec7db9e","Type":"ContainerStarted","Data":"3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54"}
Sep 17 18:31:26.901839 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:26.901808    2565 generic.go:356] "Generic (PLEG): container finished" podID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerID="3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54" exitCode=0
Sep 17 18:31:26.902266 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:26.901868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" event={"ID":"05f58ae4-1118-4c12-8630-88951ec7db9e","Type":"ContainerDied","Data":"3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54"}
Sep 17 18:31:27.908247 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:27.908211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" event={"ID":"05f58ae4-1118-4c12-8630-88951ec7db9e","Type":"ContainerStarted","Data":"96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2"}
Sep 17 18:31:27.933873 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:27.933824    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podStartSLOduration=4.153470598 podStartE2EDuration="7.933803917s" totalImagesPullingTime="1.734546412s" totalInitContainerRuntime="2.045786907s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:27.931508207 +0000 UTC m=+3030.767277432" watchObservedRunningTime="2026-09-17 18:31:27.933803917 +0000 UTC m=+3030.769573142"
Sep 17 18:31:30.954646 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:30.954613    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:30.955058 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:30.954669    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:30.956200 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:30.956173    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:31:40.955083 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:40.954984    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:31:40.967412 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:40.967370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:31:50.955552 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:31:50.955503    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:32:00.954768 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:32:00.954719    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:32:10.955569 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:32:10.955523    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:32:20.954999 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:32:20.954954    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:32:30.955162 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:32:30.955101    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:32:40.955118 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:32:40.955070    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:32:50.954831 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:32:50.954761    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:33:00.954779 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:00.954719    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:33:10.955244 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:10.955128    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Sep 17 18:33:20.969639 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:20.969607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:33:20.988715 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:20.988686    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:33:29.791033 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:29.790999    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"]
Sep 17 18:33:44.813223 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:44.813158    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main" containerID="cri-o://96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2" gracePeriod=45
Sep 17 18:33:45.434565 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.434528    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="llm-d-routing-sidecar" containerID="cri-o://d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74" gracePeriod=45
Sep 17 18:33:45.456041 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.456014    2565 generic.go:356] "Generic (PLEG): container finished" podID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerID="96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2" exitCode=0
Sep 17 18:33:45.456145 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.456084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" event={"ID":"05f58ae4-1118-4c12-8630-88951ec7db9e","Type":"ContainerDied","Data":"96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2"}
Sep 17 18:33:45.667051 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.667027    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:33:45.853848 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.853776    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-kserve-provision-location\") pod \"05f58ae4-1118-4c12-8630-88951ec7db9e\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") "
Sep 17 18:33:45.853848 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.853822    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-model-cache\") pod \"05f58ae4-1118-4c12-8630-88951ec7db9e\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") "
Sep 17 18:33:45.853848 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.853843    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-home\") pod \"05f58ae4-1118-4c12-8630-88951ec7db9e\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") "
Sep 17 18:33:45.854459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.853885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f58ae4-1118-4c12-8630-88951ec7db9e-tls-certs\") pod \"05f58ae4-1118-4c12-8630-88951ec7db9e\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") "
Sep 17 18:33:45.854459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.853931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f58ae4-1118-4c12-8630-88951ec7db9e-cabundle-cert\") pod \"05f58ae4-1118-4c12-8630-88951ec7db9e\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") "
Sep 17 18:33:45.854459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.853964    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-dshm\") pod \"05f58ae4-1118-4c12-8630-88951ec7db9e\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") "
Sep 17 18:33:45.854459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.854101    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "05f58ae4-1118-4c12-8630-88951ec7db9e" (UID: "05f58ae4-1118-4c12-8630-88951ec7db9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.854459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.854114    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfsmt\" (UniqueName: \"kubernetes.io/projected/05f58ae4-1118-4c12-8630-88951ec7db9e-kube-api-access-vfsmt\") pod \"05f58ae4-1118-4c12-8630-88951ec7db9e\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") "
Sep 17 18:33:45.854459 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.854175    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-tmp-dir\") pod \"05f58ae4-1118-4c12-8630-88951ec7db9e\" (UID: \"05f58ae4-1118-4c12-8630-88951ec7db9e\") "
Sep 17 18:33:45.854897 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.854359    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f58ae4-1118-4c12-8630-88951ec7db9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "05f58ae4-1118-4c12-8630-88951ec7db9e" (UID: "05f58ae4-1118-4c12-8630-88951ec7db9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:45.854897 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.854455    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.854897 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.854605    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-home" (OuterVolumeSpecName: "home") pod "05f58ae4-1118-4c12-8630-88951ec7db9e" (UID: "05f58ae4-1118-4c12-8630-88951ec7db9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.857026 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.856990    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-dshm" (OuterVolumeSpecName: "dshm") pod "05f58ae4-1118-4c12-8630-88951ec7db9e" (UID: "05f58ae4-1118-4c12-8630-88951ec7db9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.857152 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.857092    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f58ae4-1118-4c12-8630-88951ec7db9e-kube-api-access-vfsmt" (OuterVolumeSpecName: "kube-api-access-vfsmt") pod "05f58ae4-1118-4c12-8630-88951ec7db9e" (UID: "05f58ae4-1118-4c12-8630-88951ec7db9e"). InnerVolumeSpecName "kube-api-access-vfsmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:45.857152 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.857105    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f58ae4-1118-4c12-8630-88951ec7db9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05f58ae4-1118-4c12-8630-88951ec7db9e" (UID: "05f58ae4-1118-4c12-8630-88951ec7db9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:45.867105 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.867083    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05f58ae4-1118-4c12-8630-88951ec7db9e" (UID: "05f58ae4-1118-4c12-8630-88951ec7db9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.877346 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.877323    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05f58ae4-1118-4c12-8630-88951ec7db9e" (UID: "05f58ae4-1118-4c12-8630-88951ec7db9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.955449 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.955415    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f58ae4-1118-4c12-8630-88951ec7db9e-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.955449 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.955444    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f58ae4-1118-4c12-8630-88951ec7db9e-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.955449 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.955453    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.955677 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.955461    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfsmt\" (UniqueName: \"kubernetes.io/projected/05f58ae4-1118-4c12-8630-88951ec7db9e-kube-api-access-vfsmt\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.955677 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.955470    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.955677 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.955480    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.955677 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:45.955487    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f58ae4-1118-4c12-8630-88951ec7db9e-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:33:46.461720 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.461684    2565 generic.go:356] "Generic (PLEG): container finished" podID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerID="d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74" exitCode=0
Sep 17 18:33:46.461926 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.461731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" event={"ID":"05f58ae4-1118-4c12-8630-88951ec7db9e","Type":"ContainerDied","Data":"d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74"}
Sep 17 18:33:46.461926 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.461743    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"
Sep 17 18:33:46.461926 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.461787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8" event={"ID":"05f58ae4-1118-4c12-8630-88951ec7db9e","Type":"ContainerDied","Data":"e20aa836812b96afb4d884e38a91cefb4ac2d13ac8086b55f3f39021fa5fb826"}
Sep 17 18:33:46.461926 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.461812    2565 scope.go:117] "RemoveContainer" containerID="96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2"
Sep 17 18:33:46.470782 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.470764    2565 scope.go:117] "RemoveContainer" containerID="3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54"
Sep 17 18:33:46.478051 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.478036    2565 scope.go:117] "RemoveContainer" containerID="d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74"
Sep 17 18:33:46.484236 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.484213    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"]
Sep 17 18:33:46.485756 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.485739    2565 scope.go:117] "RemoveContainer" containerID="96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2"
Sep 17 18:33:46.486028 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:33:46.486009    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2\": container with ID starting with 96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2 not found: ID does not exist" containerID="96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2"
Sep 17 18:33:46.486079 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.486038    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2"} err="failed to get container status \"96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2\": rpc error: code = NotFound desc = could not find container \"96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2\": container with ID starting with 96241e8f1e3e23dec49bc60ae7e00f17b9f72b6b8d05f8c44806d3a2ac5b0ba2 not found: ID does not exist"
Sep 17 18:33:46.486079 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.486062    2565 scope.go:117] "RemoveContainer" containerID="3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54"
Sep 17 18:33:46.486316 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:33:46.486292    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54\": container with ID starting with 3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54 not found: ID does not exist" containerID="3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54"
Sep 17 18:33:46.486382 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.486318    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54"} err="failed to get container status \"3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54\": rpc error: code = NotFound desc = could not find container \"3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54\": container with ID starting with 3b37be4c2e6ebcf0b46bd1f413be9cdb63e85129793380b488055a9bf2bd4a54 not found: ID does not exist"
Sep 17 18:33:46.486382 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.486336    2565 scope.go:117] "RemoveContainer" containerID="d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74"
Sep 17 18:33:46.486625 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:33:46.486598    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74\": container with ID starting with d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74 not found: ID does not exist" containerID="d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74"
Sep 17 18:33:46.486733 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.486635    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74"} err="failed to get container status \"d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74\": rpc error: code = NotFound desc = could not find container \"d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74\": container with ID starting with d07ba761f15885bbf45f210b94244efe326b340ff567b879b3d929dbdf5cac74 not found: ID does not exist"
Sep 17 18:33:46.488792 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:46.488771    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-944bd467c-hhhg8"]
Sep 17 18:33:47.349021 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:33:47.348980    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" path="/var/lib/kubelet/pods/05f58ae4-1118-4c12-8630-88951ec7db9e/volumes"
Sep 17 18:34:24.953947 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:24.953909    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"]
Sep 17 18:34:24.954521 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:24.954111    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy" containerID="cri-o://e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf" gracePeriod=30
Sep 17 18:34:28.685552 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685519    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"]
Sep 17 18:34:28.685935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685774    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="storage-initializer"
Sep 17 18:34:28.685935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685784    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="storage-initializer"
Sep 17 18:34:28.685935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685792    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main"
Sep 17 18:34:28.685935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685797    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main"
Sep 17 18:34:28.685935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685806    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="llm-d-routing-sidecar"
Sep 17 18:34:28.685935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685812    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="llm-d-routing-sidecar"
Sep 17 18:34:28.685935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685860    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="llm-d-routing-sidecar"
Sep 17 18:34:28.685935 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.685868    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f58ae4-1118-4c12-8630-88951ec7db9e" containerName="main"
Sep 17 18:34:28.688744 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.688726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.691061 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.691036    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:34:28.691213 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.691103    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 18:34:28.691213 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.691047    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-6pm7x\""
Sep 17 18:34:28.691343 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.691320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 18:34:28.691428 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.691336    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 18:34:28.691428 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.691378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 18:34:28.700733 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.700714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"]
Sep 17 18:34:28.782360 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.782333    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.782527 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.782364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2e391b-a533-4d8d-bfac-36f855737c9d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.782527 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.782385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.782527 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.782503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.782643 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.782542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2e391b-a533-4d8d-bfac-36f855737c9d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.782643 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.782568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.782643 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.782615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqm5\" (UniqueName: \"kubernetes.io/projected/3f2e391b-a533-4d8d-bfac-36f855737c9d-kube-api-access-kkqm5\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.782737 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.782639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883116 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883116 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqm5\" (UniqueName: \"kubernetes.io/projected/3f2e391b-a533-4d8d-bfac-36f855737c9d-kube-api-access-kkqm5\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2e391b-a533-4d8d-bfac-36f855737c9d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883340 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2e391b-a533-4d8d-bfac-36f855737c9d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883694 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883751 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883796 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883832 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.883950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.883934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2e391b-a533-4d8d-bfac-36f855737c9d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.885389 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.885372    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.885792 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.885772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2e391b-a533-4d8d-bfac-36f855737c9d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.890453 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.890431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqm5\" (UniqueName: \"kubernetes.io/projected/3f2e391b-a533-4d8d-bfac-36f855737c9d-kube-api-access-kkqm5\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:28.998666 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:28.998579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:29.301965 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:34:29.301923    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2e391b_a533_4d8d_bfac_36f855737c9d.slice/crio-90183994db8e109c94775ae94534f14a3452f2aa1600c4ad1b5b110ff94c0926 WatchSource:0}: Error finding container 90183994db8e109c94775ae94534f14a3452f2aa1600c4ad1b5b110ff94c0926: Status 404 returned error can't find the container with id 90183994db8e109c94775ae94534f14a3452f2aa1600c4ad1b5b110ff94c0926
Sep 17 18:34:29.303313 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:29.303279    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"]
Sep 17 18:34:29.624588 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:29.624491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" event={"ID":"3f2e391b-a533-4d8d-bfac-36f855737c9d","Type":"ContainerStarted","Data":"5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0"}
Sep 17 18:34:29.624588 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:29.624571    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" event={"ID":"3f2e391b-a533-4d8d-bfac-36f855737c9d","Type":"ContainerStarted","Data":"90183994db8e109c94775ae94534f14a3452f2aa1600c4ad1b5b110ff94c0926"}
Sep 17 18:34:30.209746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.209713    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:34:30.298247 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298220    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istiod-ca-cert\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.298406 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-socket\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.298483 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298373    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxxfr\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-kube-api-access-jxxfr\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.298483 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298448    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-credential-socket\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.298607 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298481    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-data\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.298607 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298546    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-podinfo\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.298607 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298590    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:30.298766 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-certs\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.298766 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298645    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:34:30.298766 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298655    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-token\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.298766 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298721    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:30.299039 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298807    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-envoy\") pod \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\" (UID: \"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb\") "
Sep 17 18:34:30.299039 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298815    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:30.299039 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.298944    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:30.299213 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.299093    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.299213 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.299113    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.299213 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.299126    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.299213 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.299140    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.299213 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.299154    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.300975 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.300951    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:30.301143 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.301126    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:30.301217 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.301201    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:34:30.301359 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.301340    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-kube-api-access-jxxfr" (OuterVolumeSpecName: "kube-api-access-jxxfr") pod "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" (UID: "a5c9e448-fe8b-4d1d-9861-3fefb9b920cb"). InnerVolumeSpecName "kube-api-access-jxxfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:30.402425 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.400061    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxxfr\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-kube-api-access-jxxfr\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.402425 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.400099    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.402425 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.400123    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.402425 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.400136    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:34:30.630118 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.630036    2565 generic.go:356] "Generic (PLEG): container finished" podID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerID="e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf" exitCode=0
Sep 17 18:34:30.630257 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.630116    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"
Sep 17 18:34:30.630257 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.630120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" event={"ID":"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb","Type":"ContainerDied","Data":"e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf"}
Sep 17 18:34:30.630257 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.630206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz" event={"ID":"a5c9e448-fe8b-4d1d-9861-3fefb9b920cb","Type":"ContainerDied","Data":"a9d429a737a7cb9cc8fca241799e65a27f50d39bcd4867411bd3361588d70471"}
Sep 17 18:34:30.630257 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.630230    2565 scope.go:117] "RemoveContainer" containerID="e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf"
Sep 17 18:34:30.638913 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.638894    2565 scope.go:117] "RemoveContainer" containerID="e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05"
Sep 17 18:34:30.648715 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.648695    2565 scope.go:117] "RemoveContainer" containerID="e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf"
Sep 17 18:34:30.649011 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:34:30.648986    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf\": container with ID starting with e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf not found: ID does not exist" containerID="e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf"
Sep 17 18:34:30.649112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.649015    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf"} err="failed to get container status \"e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf\": rpc error: code = NotFound desc = could not find container \"e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf\": container with ID starting with e4841735965bd87e26bef98a768b7dfe17369a745503e8e6bd12e1ec1c9481bf not found: ID does not exist"
Sep 17 18:34:30.649112 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.649031    2565 scope.go:117] "RemoveContainer" containerID="e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05"
Sep 17 18:34:30.649356 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:34:30.649336    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05\": container with ID starting with e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05 not found: ID does not exist" containerID="e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05"
Sep 17 18:34:30.649465 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.649358    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05"} err="failed to get container status \"e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05\": rpc error: code = NotFound desc = could not find container \"e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05\": container with ID starting with e68d2f1e1fd7f67ba406f31409c1d0adca3af5722ad589e9578c4ffbc0d30c05 not found: ID does not exist"
Sep 17 18:34:30.652753 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.652726    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"]
Sep 17 18:34:30.657570 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:30.657520    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-gkngz"]
Sep 17 18:34:31.349063 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:31.349003    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" path="/var/lib/kubelet/pods/a5c9e448-fe8b-4d1d-9861-3fefb9b920cb/volumes"
Sep 17 18:34:32.640783 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:32.640753    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerID="5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0" exitCode=0
Sep 17 18:34:32.641168 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:32.640814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" event={"ID":"3f2e391b-a533-4d8d-bfac-36f855737c9d","Type":"ContainerDied","Data":"5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0"}
Sep 17 18:34:33.647122 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:33.647090    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" event={"ID":"3f2e391b-a533-4d8d-bfac-36f855737c9d","Type":"ContainerStarted","Data":"12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20"}
Sep 17 18:34:33.668748 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:33.668701    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podStartSLOduration=3.423071745 podStartE2EDuration="5.668684783s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.245613038s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:33.666849012 +0000 UTC m=+3216.502618235" watchObservedRunningTime="2026-09-17 18:34:33.668684783 +0000 UTC m=+3216.504454008"
Sep 17 18:34:33.998892 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:33.998852    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:38.999141 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:38.999103    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:34:39.000796 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:39.000767    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:34:48.999474 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:48.999423    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:34:58.999216 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:34:58.999161    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:35:08.999634 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:35:08.999582    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:35:18.999858 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:35:18.999807    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:35:28.999597 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:35:28.999546    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:35:38.999517 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:35:38.999471    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:35:48.999606 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:35:48.999554    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:35:58.999712 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:35:58.999670    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:35:59.493255 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:35:59.493224    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:35:59.496574 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:35:59.496551    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:36:08.999776 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:36:08.999726    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:36:18.999949 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:36:18.999907    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:36:29.009574 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:36:29.009541    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:36:29.017465 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:36:29.017439    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:36:45.102173 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:36:45.102140    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"]
Sep 17 18:37:00.124834 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:00.124792    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main" containerID="cri-o://12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20" gracePeriod=45
Sep 17 18:37:00.855362 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:00.855335    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:37:01.027498 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027464    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2e391b-a533-4d8d-bfac-36f855737c9d-cabundle-cert\") pod \"3f2e391b-a533-4d8d-bfac-36f855737c9d\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") "
Sep 17 18:37:01.027681 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027509    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-kserve-provision-location\") pod \"3f2e391b-a533-4d8d-bfac-36f855737c9d\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") "
Sep 17 18:37:01.027681 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027552    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-home\") pod \"3f2e391b-a533-4d8d-bfac-36f855737c9d\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") "
Sep 17 18:37:01.027681 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-model-cache\") pod \"3f2e391b-a533-4d8d-bfac-36f855737c9d\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") "
Sep 17 18:37:01.027681 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027589    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-dshm\") pod \"3f2e391b-a533-4d8d-bfac-36f855737c9d\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") "
Sep 17 18:37:01.027681 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-tmp-dir\") pod \"3f2e391b-a533-4d8d-bfac-36f855737c9d\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") "
Sep 17 18:37:01.027968 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027708    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkqm5\" (UniqueName: \"kubernetes.io/projected/3f2e391b-a533-4d8d-bfac-36f855737c9d-kube-api-access-kkqm5\") pod \"3f2e391b-a533-4d8d-bfac-36f855737c9d\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") "
Sep 17 18:37:01.027968 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2e391b-a533-4d8d-bfac-36f855737c9d-tls-certs\") pod \"3f2e391b-a533-4d8d-bfac-36f855737c9d\" (UID: \"3f2e391b-a533-4d8d-bfac-36f855737c9d\") "
Sep 17 18:37:01.027968 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027844    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f2e391b-a533-4d8d-bfac-36f855737c9d" (UID: "3f2e391b-a533-4d8d-bfac-36f855737c9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:01.027968 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.027935    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2e391b-a533-4d8d-bfac-36f855737c9d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f2e391b-a533-4d8d-bfac-36f855737c9d" (UID: "3f2e391b-a533-4d8d-bfac-36f855737c9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:01.028178 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.028062    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2e391b-a533-4d8d-bfac-36f855737c9d-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.028178 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.028084    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.028353 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.028313    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-home" (OuterVolumeSpecName: "home") pod "3f2e391b-a533-4d8d-bfac-36f855737c9d" (UID: "3f2e391b-a533-4d8d-bfac-36f855737c9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:01.030237 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.030208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-dshm" (OuterVolumeSpecName: "dshm") pod "3f2e391b-a533-4d8d-bfac-36f855737c9d" (UID: "3f2e391b-a533-4d8d-bfac-36f855737c9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:01.030480 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.030457    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2e391b-a533-4d8d-bfac-36f855737c9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f2e391b-a533-4d8d-bfac-36f855737c9d" (UID: "3f2e391b-a533-4d8d-bfac-36f855737c9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:01.030643 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.030620    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2e391b-a533-4d8d-bfac-36f855737c9d-kube-api-access-kkqm5" (OuterVolumeSpecName: "kube-api-access-kkqm5") pod "3f2e391b-a533-4d8d-bfac-36f855737c9d" (UID: "3f2e391b-a533-4d8d-bfac-36f855737c9d"). InnerVolumeSpecName "kube-api-access-kkqm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:01.041159 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.041134    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f2e391b-a533-4d8d-bfac-36f855737c9d" (UID: "3f2e391b-a533-4d8d-bfac-36f855737c9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:01.052212 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.052190    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f2e391b-a533-4d8d-bfac-36f855737c9d" (UID: "3f2e391b-a533-4d8d-bfac-36f855737c9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:01.128661 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.128628    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.128661 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.128656    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.128661 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.128665    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.129079 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.128675    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkqm5\" (UniqueName: \"kubernetes.io/projected/3f2e391b-a533-4d8d-bfac-36f855737c9d-kube-api-access-kkqm5\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.129079 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.128684    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2e391b-a533-4d8d-bfac-36f855737c9d-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.129079 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.128693    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2e391b-a533-4d8d-bfac-36f855737c9d-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.206437 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.206388    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerID="12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20" exitCode=0
Sep 17 18:37:01.206625 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.206459    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"
Sep 17 18:37:01.206625 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.206472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" event={"ID":"3f2e391b-a533-4d8d-bfac-36f855737c9d","Type":"ContainerDied","Data":"12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20"}
Sep 17 18:37:01.206625 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.206527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh" event={"ID":"3f2e391b-a533-4d8d-bfac-36f855737c9d","Type":"ContainerDied","Data":"90183994db8e109c94775ae94534f14a3452f2aa1600c4ad1b5b110ff94c0926"}
Sep 17 18:37:01.206625 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.206549    2565 scope.go:117] "RemoveContainer" containerID="12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20"
Sep 17 18:37:01.220803 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.220780    2565 scope.go:117] "RemoveContainer" containerID="5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0"
Sep 17 18:37:01.228024 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.228008    2565 scope.go:117] "RemoveContainer" containerID="12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20"
Sep 17 18:37:01.228277 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:37:01.228259    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20\": container with ID starting with 12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20 not found: ID does not exist" containerID="12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20"
Sep 17 18:37:01.228330 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.228285    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20"} err="failed to get container status \"12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20\": rpc error: code = NotFound desc = could not find container \"12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20\": container with ID starting with 12671f8e8d284fdbe9f7de5bd322e461ef862cd743f3c46177e5e0b464cbfe20 not found: ID does not exist"
Sep 17 18:37:01.228376 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.228334    2565 scope.go:117] "RemoveContainer" containerID="5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0"
Sep 17 18:37:01.228622 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:37:01.228595    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0\": container with ID starting with 5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0 not found: ID does not exist" containerID="5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0"
Sep 17 18:37:01.228708 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.228628    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0"} err="failed to get container status \"5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0\": rpc error: code = NotFound desc = could not find container \"5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0\": container with ID starting with 5f31e4e62a9c0605176effb5fbc2ce8e311db1acc377ee8760a2058b5d4627b0 not found: ID does not exist"
Sep 17 18:37:01.230377 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.230356    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"]
Sep 17 18:37:01.233853 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.233834    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d5c8f5446jv7rh"]
Sep 17 18:37:01.348088 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:01.348024    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" path="/var/lib/kubelet/pods/3f2e391b-a533-4d8d-bfac-36f855737c9d/volumes"
Sep 17 18:37:22.637114 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637077    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"]
Sep 17 18:37:22.637808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637353    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy"
Sep 17 18:37:22.637808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637366    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy"
Sep 17 18:37:22.637808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637379    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="storage-initializer"
Sep 17 18:37:22.637808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637385    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="storage-initializer"
Sep 17 18:37:22.637808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637414    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main"
Sep 17 18:37:22.637808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637419    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main"
Sep 17 18:37:22.637808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637464    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy"
Sep 17 18:37:22.637808 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.637472    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2e391b-a533-4d8d-bfac-36f855737c9d" containerName="main"
Sep 17 18:37:22.640334 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.640318    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:22.643025 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.642998    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 18:37:22.643162 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.643142    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 18:37:22.643241 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.643166    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-4994h\""
Sep 17 18:37:22.643299 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.643248    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 18:37:22.643509 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.643486    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-gspqb\""
Sep 17 18:37:22.650090 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.650066    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"]
Sep 17 18:37:22.681457 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.681427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38363f48-5d9d-477b-b143-2b6470a7262d-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn\" (UID: \"38363f48-5d9d-477b-b143-2b6470a7262d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:22.681596 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.681494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpft7\" (UniqueName: \"kubernetes.io/projected/38363f48-5d9d-477b-b143-2b6470a7262d-kube-api-access-mpft7\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn\" (UID: \"38363f48-5d9d-477b-b143-2b6470a7262d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:22.700092 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.700063    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:37:22.700337 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.700325    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy"
Sep 17 18:37:22.700380 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.700340    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy"
Sep 17 18:37:22.700443 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.700407    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c9e448-fe8b-4d1d-9861-3fefb9b920cb" containerName="istio-proxy"
Sep 17 18:37:22.703188 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.703172    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.705331 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.705315    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-ndff7\""
Sep 17 18:37:22.705415 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.705371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:37:22.705470 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.705319    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 18:37:22.712852 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.712832    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:37:22.782185 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38363f48-5d9d-477b-b143-2b6470a7262d-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn\" (UID: \"38363f48-5d9d-477b-b143-2b6470a7262d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:22.782347 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.782347 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/019c36b6-9b5f-4bca-8f78-fe860e5007d2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.782347 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.782347 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.782556 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.782556 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.782556 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782475    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.782556 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpft7\" (UniqueName: \"kubernetes.io/projected/38363f48-5d9d-477b-b143-2b6470a7262d-kube-api-access-mpft7\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn\" (UID: \"38363f48-5d9d-477b-b143-2b6470a7262d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:22.782556 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.782534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pc8\" (UniqueName: \"kubernetes.io/projected/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kube-api-access-h6pc8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.784852 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.784831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38363f48-5d9d-477b-b143-2b6470a7262d-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn\" (UID: \"38363f48-5d9d-477b-b143-2b6470a7262d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:22.790462 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.790438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpft7\" (UniqueName: \"kubernetes.io/projected/38363f48-5d9d-477b-b143-2b6470a7262d-kube-api-access-mpft7\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn\" (UID: \"38363f48-5d9d-477b-b143-2b6470a7262d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pc8\" (UniqueName: \"kubernetes.io/projected/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kube-api-access-h6pc8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/019c36b6-9b5f-4bca-8f78-fe860e5007d2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.884883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.887685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.888948 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.887924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.889535 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.889022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.889535 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.889321    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.889844 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.889821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/019c36b6-9b5f-4bca-8f78-fe860e5007d2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.890137 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.890118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.896032 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.896012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pc8\" (UniqueName: \"kubernetes.io/projected/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kube-api-access-h6pc8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:22.950880 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:22.950856    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:23.013811 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:23.013779    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:37:23.222147 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:23.222114    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"]
Sep 17 18:37:23.225465 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:37:23.225435    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38363f48_5d9d_477b_b143_2b6470a7262d.slice/crio-874b1eae99567c0bd0b54cce70c799dbc37cacfb782fcc5ac03989ae6324e606 WatchSource:0}: Error finding container 874b1eae99567c0bd0b54cce70c799dbc37cacfb782fcc5ac03989ae6324e606: Status 404 returned error can't find the container with id 874b1eae99567c0bd0b54cce70c799dbc37cacfb782fcc5ac03989ae6324e606
Sep 17 18:37:23.284828 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:23.284793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" event={"ID":"38363f48-5d9d-477b-b143-2b6470a7262d","Type":"ContainerStarted","Data":"874b1eae99567c0bd0b54cce70c799dbc37cacfb782fcc5ac03989ae6324e606"}
Sep 17 18:37:23.292570 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:23.292539    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:37:23.297107 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:37:23.297083    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019c36b6_9b5f_4bca_8f78_fe860e5007d2.slice/crio-fa30cf9f100ce1816371b96674b7d7901e40a4c5d3bc40aa19a667acf7a69dff WatchSource:0}: Error finding container fa30cf9f100ce1816371b96674b7d7901e40a4c5d3bc40aa19a667acf7a69dff: Status 404 returned error can't find the container with id fa30cf9f100ce1816371b96674b7d7901e40a4c5d3bc40aa19a667acf7a69dff
Sep 17 18:37:24.289980 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:24.289946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"019c36b6-9b5f-4bca-8f78-fe860e5007d2","Type":"ContainerStarted","Data":"a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70"}
Sep 17 18:37:24.290453 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:24.290011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"019c36b6-9b5f-4bca-8f78-fe860e5007d2","Type":"ContainerStarted","Data":"fa30cf9f100ce1816371b96674b7d7901e40a4c5d3bc40aa19a667acf7a69dff"}
Sep 17 18:37:24.291547 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:24.291521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" event={"ID":"38363f48-5d9d-477b-b143-2b6470a7262d","Type":"ContainerStarted","Data":"6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3"}
Sep 17 18:37:24.291643 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:24.291569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:37:24.324133 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:24.324086    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" podStartSLOduration=2.324073577 podStartE2EDuration="2.324073577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:24.323006299 +0000 UTC m=+3387.158775534" watchObservedRunningTime="2026-09-17 18:37:24.324073577 +0000 UTC m=+3387.159842828"
Sep 17 18:37:26.302914 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:26.302882    2565 generic.go:356] "Generic (PLEG): container finished" podID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" containerID="a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70" exitCode=0
Sep 17 18:37:26.303300 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:26.302953    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"019c36b6-9b5f-4bca-8f78-fe860e5007d2","Type":"ContainerDied","Data":"a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70"}
Sep 17 18:37:27.310290 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:27.310246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"019c36b6-9b5f-4bca-8f78-fe860e5007d2","Type":"ContainerStarted","Data":"2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673"}
Sep 17 18:37:27.330317 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:27.330270    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.174716145 podStartE2EDuration="5.330255559s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.155539414s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:27.328039656 +0000 UTC m=+3390.163808879" watchObservedRunningTime="2026-09-17 18:37:27.330255559 +0000 UTC m=+3390.166024782"
Sep 17 18:37:55.299934 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:37:55.299849    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:39:07.781854 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:07.781818    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"]
Sep 17 18:39:15.299149 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:15.299070    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:22.801471 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:22.801389    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" containerName="main" containerID="cri-o://6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3" gracePeriod=45
Sep 17 18:39:24.357069 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:24.357031    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:39:25.299426 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:25.299375    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:35.300027 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:35.299987    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:35.300499 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:35.300091    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:39:39.378819 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:39.378781    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" containerName="main" containerID="cri-o://2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673" gracePeriod=45
Sep 17 18:39:40.116022 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.115991    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:39:40.190455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.190429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6pc8\" (UniqueName: \"kubernetes.io/projected/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kube-api-access-h6pc8\") pod \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") "
Sep 17 18:39:40.190627 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.190467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kserve-provision-location\") pod \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") "
Sep 17 18:39:40.190627 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.190497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-dshm\") pod \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") "
Sep 17 18:39:40.190627 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.190535    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tls-certs\") pod \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") "
Sep 17 18:39:40.190627 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.190567    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/019c36b6-9b5f-4bca-8f78-fe860e5007d2-cabundle-cert\") pod \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") "
Sep 17 18:39:40.190627 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.190594    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-home\") pod \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") "
Sep 17 18:39:40.190918 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.190632    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-model-cache\") pod \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") "
Sep 17 18:39:40.190918 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.190662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tmp-dir\") pod \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\" (UID: \"019c36b6-9b5f-4bca-8f78-fe860e5007d2\") "
Sep 17 18:39:40.191104 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.191051    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019c36b6-9b5f-4bca-8f78-fe860e5007d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "019c36b6-9b5f-4bca-8f78-fe860e5007d2" (UID: "019c36b6-9b5f-4bca-8f78-fe860e5007d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:39:40.191182 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.191124    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "019c36b6-9b5f-4bca-8f78-fe860e5007d2" (UID: "019c36b6-9b5f-4bca-8f78-fe860e5007d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:40.191669 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.191641    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-home" (OuterVolumeSpecName: "home") pod "019c36b6-9b5f-4bca-8f78-fe860e5007d2" (UID: "019c36b6-9b5f-4bca-8f78-fe860e5007d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:40.192990 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.192967    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "019c36b6-9b5f-4bca-8f78-fe860e5007d2" (UID: "019c36b6-9b5f-4bca-8f78-fe860e5007d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:39:40.193116 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.193091    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kube-api-access-h6pc8" (OuterVolumeSpecName: "kube-api-access-h6pc8") pod "019c36b6-9b5f-4bca-8f78-fe860e5007d2" (UID: "019c36b6-9b5f-4bca-8f78-fe860e5007d2"). InnerVolumeSpecName "kube-api-access-h6pc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:40.193217 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.193151    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-dshm" (OuterVolumeSpecName: "dshm") pod "019c36b6-9b5f-4bca-8f78-fe860e5007d2" (UID: "019c36b6-9b5f-4bca-8f78-fe860e5007d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:40.208457 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.208430    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "019c36b6-9b5f-4bca-8f78-fe860e5007d2" (UID: "019c36b6-9b5f-4bca-8f78-fe860e5007d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:40.220070 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.220048    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "019c36b6-9b5f-4bca-8f78-fe860e5007d2" (UID: "019c36b6-9b5f-4bca-8f78-fe860e5007d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:40.291526 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.291483    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pc8\" (UniqueName: \"kubernetes.io/projected/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kube-api-access-h6pc8\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:40.291526 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.291516    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:40.291526 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.291527    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:40.291748 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.291536    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:40.291748 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.291545    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/019c36b6-9b5f-4bca-8f78-fe860e5007d2-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:40.291748 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.291552    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:40.291748 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.291559    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:40.291748 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.291567    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c36b6-9b5f-4bca-8f78-fe860e5007d2-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:40.810570 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.810535    2565 generic.go:356] "Generic (PLEG): container finished" podID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" containerID="2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673" exitCode=0
Sep 17 18:39:40.810960 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.810587    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:39:40.810960 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.810624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"019c36b6-9b5f-4bca-8f78-fe860e5007d2","Type":"ContainerDied","Data":"2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673"}
Sep 17 18:39:40.810960 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.810676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"019c36b6-9b5f-4bca-8f78-fe860e5007d2","Type":"ContainerDied","Data":"fa30cf9f100ce1816371b96674b7d7901e40a4c5d3bc40aa19a667acf7a69dff"}
Sep 17 18:39:40.810960 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.810698    2565 scope.go:117] "RemoveContainer" containerID="2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673"
Sep 17 18:39:40.820580 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.820500    2565 scope.go:117] "RemoveContainer" containerID="a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70"
Sep 17 18:39:40.828938 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.828921    2565 scope.go:117] "RemoveContainer" containerID="2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673"
Sep 17 18:39:40.829201 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:39:40.829182    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673\": container with ID starting with 2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673 not found: ID does not exist" containerID="2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673"
Sep 17 18:39:40.829245 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.829208    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673"} err="failed to get container status \"2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673\": rpc error: code = NotFound desc = could not find container \"2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673\": container with ID starting with 2ed019ecdacf047a7c72ff61bc5968b2c9b30700b4bef641e3f07c72ced40673 not found: ID does not exist"
Sep 17 18:39:40.829245 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.829224    2565 scope.go:117] "RemoveContainer" containerID="a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70"
Sep 17 18:39:40.829488 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:39:40.829470    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70\": container with ID starting with a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70 not found: ID does not exist" containerID="a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70"
Sep 17 18:39:40.829542 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.829493    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70"} err="failed to get container status \"a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70\": rpc error: code = NotFound desc = could not find container \"a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70\": container with ID starting with a3e4f8565f85cb852b48bd7665b5df857ff71cec2427623f138422da5d117b70 not found: ID does not exist"
Sep 17 18:39:40.833289 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.833270    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:39:40.836591 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:40.836571    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:39:41.349015 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:41.348984    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" path="/var/lib/kubelet/pods/019c36b6-9b5f-4bca-8f78-fe860e5007d2/volumes"
Sep 17 18:39:45.299278 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:45.299235    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:53.029780 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.029752    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:39:53.193646 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.193617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpft7\" (UniqueName: \"kubernetes.io/projected/38363f48-5d9d-477b-b143-2b6470a7262d-kube-api-access-mpft7\") pod \"38363f48-5d9d-477b-b143-2b6470a7262d\" (UID: \"38363f48-5d9d-477b-b143-2b6470a7262d\") "
Sep 17 18:39:53.193818 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.193675    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38363f48-5d9d-477b-b143-2b6470a7262d-tls-certs\") pod \"38363f48-5d9d-477b-b143-2b6470a7262d\" (UID: \"38363f48-5d9d-477b-b143-2b6470a7262d\") "
Sep 17 18:39:53.195744 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.195715    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38363f48-5d9d-477b-b143-2b6470a7262d-kube-api-access-mpft7" (OuterVolumeSpecName: "kube-api-access-mpft7") pod "38363f48-5d9d-477b-b143-2b6470a7262d" (UID: "38363f48-5d9d-477b-b143-2b6470a7262d"). InnerVolumeSpecName "kube-api-access-mpft7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:53.195855 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.195816    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38363f48-5d9d-477b-b143-2b6470a7262d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38363f48-5d9d-477b-b143-2b6470a7262d" (UID: "38363f48-5d9d-477b-b143-2b6470a7262d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:39:53.294509 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.294473    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpft7\" (UniqueName: \"kubernetes.io/projected/38363f48-5d9d-477b-b143-2b6470a7262d-kube-api-access-mpft7\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:53.294509 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.294507    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38363f48-5d9d-477b-b143-2b6470a7262d-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:39:53.857862 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.857824    2565 generic.go:356] "Generic (PLEG): container finished" podID="38363f48-5d9d-477b-b143-2b6470a7262d" containerID="6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3" exitCode=0
Sep 17 18:39:53.858077 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.857870    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"
Sep 17 18:39:53.858077 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.857900    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" event={"ID":"38363f48-5d9d-477b-b143-2b6470a7262d","Type":"ContainerDied","Data":"6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3"}
Sep 17 18:39:53.858077 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.857951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn" event={"ID":"38363f48-5d9d-477b-b143-2b6470a7262d","Type":"ContainerDied","Data":"874b1eae99567c0bd0b54cce70c799dbc37cacfb782fcc5ac03989ae6324e606"}
Sep 17 18:39:53.858077 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.857974    2565 scope.go:117] "RemoveContainer" containerID="6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3"
Sep 17 18:39:53.867719 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.867698    2565 scope.go:117] "RemoveContainer" containerID="6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3"
Sep 17 18:39:53.868002 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:39:53.867979    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3\": container with ID starting with 6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3 not found: ID does not exist" containerID="6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3"
Sep 17 18:39:53.868080 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.868011    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3"} err="failed to get container status \"6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3\": rpc error: code = NotFound desc = could not find container \"6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3\": container with ID starting with 6e73545863cf4b35fa1a7ac0425b1b71e3ff38015806fd72fddd72a033760dd3 not found: ID does not exist"
Sep 17 18:39:53.873544 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.873520    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"]
Sep 17 18:39:53.876569 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:53.876550    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew88sn"]
Sep 17 18:39:55.347842 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:39:55.347807    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" path="/var/lib/kubelet/pods/38363f48-5d9d-477b-b143-2b6470a7262d/volumes"
Sep 17 18:40:05.582659 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.582622    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"]
Sep 17 18:40:05.583111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.582894    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" containerName="main"
Sep 17 18:40:05.583111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.582907    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" containerName="main"
Sep 17 18:40:05.583111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.582921    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" containerName="storage-initializer"
Sep 17 18:40:05.583111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.582927    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" containerName="storage-initializer"
Sep 17 18:40:05.583111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.582939    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" containerName="main"
Sep 17 18:40:05.583111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.582944    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" containerName="main"
Sep 17 18:40:05.583111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.582992    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="019c36b6-9b5f-4bca-8f78-fe860e5007d2" containerName="main"
Sep 17 18:40:05.583111 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.583001    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38363f48-5d9d-477b-b143-2b6470a7262d" containerName="main"
Sep 17 18:40:05.584773 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.584753    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.586750 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.586728    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 18:40:05.587359 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.587318    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 18:40:05.587455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.587437    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-7mddk\""
Sep 17 18:40:05.587527 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.587491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 18:40:05.596770 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.596749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"]
Sep 17 18:40:05.686695 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.686661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.686856 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.686723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.686856 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.686750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhqv\" (UniqueName: \"kubernetes.io/projected/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-kube-api-access-sdhqv\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.686856 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.686770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.686856 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.686786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.687006 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.686875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.787650 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.787604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.787845 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.787682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.788133 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.788106    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.788260 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.788180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.788260 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.788221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhqv\" (UniqueName: \"kubernetes.io/projected/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-kube-api-access-sdhqv\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.788260 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.788249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.788445 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.788269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.788608 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.788586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.788742 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.788715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.789534 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.789512    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"]
Sep 17 18:40:05.790564 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.790541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.790807 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.790791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.791929 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.791912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:05.794098 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.794075    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-b7vwl\""
Sep 17 18:40:05.796696 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.796672    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhqv\" (UniqueName: \"kubernetes.io/projected/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-kube-api-access-sdhqv\") pod \"scheduler-inline-config-test-kserve-8655bc757-fpn7t\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.801745 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.801726    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"]
Sep 17 18:40:05.888733 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.888655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s68r\" (UniqueName: \"kubernetes.io/projected/c9e36473-b5b9-4093-8213-60f41c7315fb-kube-api-access-4s68r\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp\" (UID: \"c9e36473-b5b9-4093-8213-60f41c7315fb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:05.888733 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.888717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e36473-b5b9-4093-8213-60f41c7315fb-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp\" (UID: \"c9e36473-b5b9-4093-8213-60f41c7315fb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:05.894604 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.894581    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:05.989354 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.989322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e36473-b5b9-4093-8213-60f41c7315fb-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp\" (UID: \"c9e36473-b5b9-4093-8213-60f41c7315fb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:05.989537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.989371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s68r\" (UniqueName: \"kubernetes.io/projected/c9e36473-b5b9-4093-8213-60f41c7315fb-kube-api-access-4s68r\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp\" (UID: \"c9e36473-b5b9-4093-8213-60f41c7315fb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:05.991926 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.991904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e36473-b5b9-4093-8213-60f41c7315fb-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp\" (UID: \"c9e36473-b5b9-4093-8213-60f41c7315fb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:05.996689 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:05.996661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s68r\" (UniqueName: \"kubernetes.io/projected/c9e36473-b5b9-4093-8213-60f41c7315fb-kube-api-access-4s68r\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp\" (UID: \"c9e36473-b5b9-4093-8213-60f41c7315fb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:06.113371 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.113332    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:06.164162 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.163830    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"]
Sep 17 18:40:06.169828 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:40:06.169786    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e7c1a7_8e08_4d17_baf9_0c8ffccd6999.slice/crio-6a46cb6ee3ff07892c107461c65e3d062656167ac879f81ca377ea558c31ccdb WatchSource:0}: Error finding container 6a46cb6ee3ff07892c107461c65e3d062656167ac879f81ca377ea558c31ccdb: Status 404 returned error can't find the container with id 6a46cb6ee3ff07892c107461c65e3d062656167ac879f81ca377ea558c31ccdb
Sep 17 18:40:06.382651 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.382504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"]
Sep 17 18:40:06.385206 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:40:06.385173    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e36473_b5b9_4093_8213_60f41c7315fb.slice/crio-df519b99231c1dcd8ed9f0006d6594f998ded65d80065bd1be9567257589a0e9 WatchSource:0}: Error finding container df519b99231c1dcd8ed9f0006d6594f998ded65d80065bd1be9567257589a0e9: Status 404 returned error can't find the container with id df519b99231c1dcd8ed9f0006d6594f998ded65d80065bd1be9567257589a0e9
Sep 17 18:40:06.909552 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.909509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" event={"ID":"c9e36473-b5b9-4093-8213-60f41c7315fb","Type":"ContainerStarted","Data":"73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4"}
Sep 17 18:40:06.910199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.910099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" event={"ID":"c9e36473-b5b9-4093-8213-60f41c7315fb","Type":"ContainerStarted","Data":"df519b99231c1dcd8ed9f0006d6594f998ded65d80065bd1be9567257589a0e9"}
Sep 17 18:40:06.910199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.910147    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:06.913001 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.912511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" event={"ID":"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999","Type":"ContainerStarted","Data":"91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d"}
Sep 17 18:40:06.913001 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.912558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" event={"ID":"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999","Type":"ContainerStarted","Data":"6a46cb6ee3ff07892c107461c65e3d062656167ac879f81ca377ea558c31ccdb"}
Sep 17 18:40:06.929441 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.929371    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" podStartSLOduration=1.9293519 podStartE2EDuration="1.9293519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:06.927715803 +0000 UTC m=+3549.763485029" watchObservedRunningTime="2026-09-17 18:40:06.9293519 +0000 UTC m=+3549.765121126"
Sep 17 18:40:06.945023 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:06.944965    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" podStartSLOduration=1.94494715 podStartE2EDuration="1.94494715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:06.944611367 +0000 UTC m=+3549.780380591" watchObservedRunningTime="2026-09-17 18:40:06.94494715 +0000 UTC m=+3549.780716374"
Sep 17 18:40:07.895618 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:07.895577    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:15.895113 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:15.895068    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:15.907349 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:15.907321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:15.956117 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:15.956087    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:37.917985 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:37.917953    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:40:39.638564 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:39.638533    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"]
Sep 17 18:40:39.646417 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:39.646370    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"]
Sep 17 18:40:39.648365 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:40:39.648320    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:40:39.648365 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:39.648365 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:39.648365 ip-10-0-129-186 kubenswrapper[2565]:  > containerID="91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d" cmd=["/bin/sleep","15"]
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:39.648379    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" message="" err=<
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]:  >
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:40:39.648419    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" podUID="e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" containerName="main" containerID="cri-o://91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d"
Sep 17 18:40:39.648681 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:39.648471    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" podUID="e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" containerName="main" containerID="cri-o://91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d" gracePeriod=60
Sep 17 18:40:39.993169 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:39.993140    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:40.036077 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.036045    2565 generic.go:356] "Generic (PLEG): container finished" podID="e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" containerID="91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d" exitCode=0
Sep 17 18:40:40.036249 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.036086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" event={"ID":"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999","Type":"ContainerDied","Data":"91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d"}
Sep 17 18:40:40.036249 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.036089    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"
Sep 17 18:40:40.036249 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.036135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" event={"ID":"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999","Type":"ContainerDied","Data":"6a46cb6ee3ff07892c107461c65e3d062656167ac879f81ca377ea558c31ccdb"}
Sep 17 18:40:40.036249 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.036160    2565 scope.go:117] "RemoveContainer" containerID="91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d"
Sep 17 18:40:40.044266 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.044251    2565 scope.go:117] "RemoveContainer" containerID="91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d"
Sep 17 18:40:40.044571 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:40:40.044547    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d\": container with ID starting with 91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d not found: ID does not exist" containerID="91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d"
Sep 17 18:40:40.044664 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.044577    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d"} err="failed to get container status \"91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d\": rpc error: code = NotFound desc = could not find container \"91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d\": container with ID starting with 91453e34fb02a462f7266cab7784f3022832931248e1de232fe92a66efebd00d not found: ID does not exist"
Sep 17 18:40:40.069236 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069211    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tls-certs\") pod \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") "
Sep 17 18:40:40.069335 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdhqv\" (UniqueName: \"kubernetes.io/projected/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-kube-api-access-sdhqv\") pod \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") "
Sep 17 18:40:40.069335 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069304    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-model-cache\") pod \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") "
Sep 17 18:40:40.069335 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069329    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-home\") pod \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") "
Sep 17 18:40:40.069515 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069348    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tmp-dir\") pod \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") "
Sep 17 18:40:40.069515 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069376    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-dshm\") pod \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\" (UID: \"e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999\") "
Sep 17 18:40:40.069630 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069605    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-home" (OuterVolumeSpecName: "home") pod "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" (UID: "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:40.069630 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" (UID: "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:40.069748 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.069634    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" (UID: "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:40.071572 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.071546    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-dshm" (OuterVolumeSpecName: "dshm") pod "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" (UID: "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:40.071664 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.071606    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" (UID: "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:40.071664 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.071615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-kube-api-access-sdhqv" (OuterVolumeSpecName: "kube-api-access-sdhqv") pod "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" (UID: "e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999"). InnerVolumeSpecName "kube-api-access-sdhqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:40.169928 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.169868    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:40:40.169928 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.169893    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdhqv\" (UniqueName: \"kubernetes.io/projected/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-kube-api-access-sdhqv\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:40:40.169928 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.169907    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:40:40.169928 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.169920    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:40:40.169928 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.169929    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:40:40.170164 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.169937    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:40:40.358749 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.358715    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"]
Sep 17 18:40:40.362049 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.362019    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t"]
Sep 17 18:40:40.946403 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:40.946335    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-fpn7t" podUID="e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Sep 17 18:40:41.348894 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:41.348789    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" path="/var/lib/kubelet/pods/e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999/volumes"
Sep 17 18:40:47.918884 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:47.918847    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:40:54.665796 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:54.665734    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerName="main" containerID="cri-o://73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4" gracePeriod=45
Sep 17 18:40:57.918337 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:57.918300    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:40:59.515209 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:59.515172    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:40:59.519691 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:40:59.519666    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:41:07.918509 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:07.918464    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:07.918900 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:07.918574    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:41:17.126168 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.126134    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"]
Sep 17 18:41:17.126576 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.126416    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" containerName="main"
Sep 17 18:41:17.126576 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.126440    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" containerName="main"
Sep 17 18:41:17.126576 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.126502    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e7c1a7-8e08-4d17-baf9-0c8ffccd6999" containerName="main"
Sep 17 18:41:17.129324 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.129304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:17.132225 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.132201    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4t7nz\""
Sep 17 18:41:17.132455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.132271    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-c6z7s\""
Sep 17 18:41:17.132455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.132299    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:41:17.132455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.132302    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:41:17.132455 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.132417    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:41:17.139577 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.139558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"]
Sep 17 18:41:17.146486 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.146459    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c36295d-ff75-4328-9633-03d5678134c3-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd\" (UID: \"0c36295d-ff75-4328-9633-03d5678134c3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:17.146585 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.146508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrrr\" (UniqueName: \"kubernetes.io/projected/0c36295d-ff75-4328-9633-03d5678134c3-kube-api-access-rzrrr\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd\" (UID: \"0c36295d-ff75-4328-9633-03d5678134c3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:17.246896 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.246871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c36295d-ff75-4328-9633-03d5678134c3-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd\" (UID: \"0c36295d-ff75-4328-9633-03d5678134c3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:17.247040 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.246902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrrr\" (UniqueName: \"kubernetes.io/projected/0c36295d-ff75-4328-9633-03d5678134c3-kube-api-access-rzrrr\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd\" (UID: \"0c36295d-ff75-4328-9633-03d5678134c3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:17.249354 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.249327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c36295d-ff75-4328-9633-03d5678134c3-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd\" (UID: \"0c36295d-ff75-4328-9633-03d5678134c3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:17.255350 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.255323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrrr\" (UniqueName: \"kubernetes.io/projected/0c36295d-ff75-4328-9633-03d5678134c3-kube-api-access-rzrrr\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd\" (UID: \"0c36295d-ff75-4328-9633-03d5678134c3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:17.440287 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.440241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:17.715016 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.714878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"]
Sep 17 18:41:17.721148 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:41:17.721115    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c36295d_ff75_4328_9633_03d5678134c3.slice/crio-56c4b5148de6a666f232c81ab95e699731da7e0c766b81c87cebd3b94f38eb4f WatchSource:0}: Error finding container 56c4b5148de6a666f232c81ab95e699731da7e0c766b81c87cebd3b94f38eb4f: Status 404 returned error can't find the container with id 56c4b5148de6a666f232c81ab95e699731da7e0c766b81c87cebd3b94f38eb4f
Sep 17 18:41:17.918989 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:17.918943    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:18.175952 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:18.175910    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" event={"ID":"0c36295d-ff75-4328-9633-03d5678134c3","Type":"ContainerStarted","Data":"29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89"}
Sep 17 18:41:18.176422 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:18.175980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" event={"ID":"0c36295d-ff75-4328-9633-03d5678134c3","Type":"ContainerStarted","Data":"56c4b5148de6a666f232c81ab95e699731da7e0c766b81c87cebd3b94f38eb4f"}
Sep 17 18:41:18.176422 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:18.176021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:41:18.192326 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:18.192282    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" podStartSLOduration=1.192268782 podStartE2EDuration="1.192268782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:18.19057495 +0000 UTC m=+3621.026344187" watchObservedRunningTime="2026-09-17 18:41:18.192268782 +0000 UTC m=+3621.028038004"
Sep 17 18:41:24.901859 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:24.901836    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:41:25.006559 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.006477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e36473-b5b9-4093-8213-60f41c7315fb-tls-certs\") pod \"c9e36473-b5b9-4093-8213-60f41c7315fb\" (UID: \"c9e36473-b5b9-4093-8213-60f41c7315fb\") "
Sep 17 18:41:25.006559 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.006539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s68r\" (UniqueName: \"kubernetes.io/projected/c9e36473-b5b9-4093-8213-60f41c7315fb-kube-api-access-4s68r\") pod \"c9e36473-b5b9-4093-8213-60f41c7315fb\" (UID: \"c9e36473-b5b9-4093-8213-60f41c7315fb\") "
Sep 17 18:41:25.008701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.008676    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e36473-b5b9-4093-8213-60f41c7315fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9e36473-b5b9-4093-8213-60f41c7315fb" (UID: "c9e36473-b5b9-4093-8213-60f41c7315fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:25.008799 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.008696    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e36473-b5b9-4093-8213-60f41c7315fb-kube-api-access-4s68r" (OuterVolumeSpecName: "kube-api-access-4s68r") pod "c9e36473-b5b9-4093-8213-60f41c7315fb" (UID: "c9e36473-b5b9-4093-8213-60f41c7315fb"). InnerVolumeSpecName "kube-api-access-4s68r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:25.107504 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.107475    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e36473-b5b9-4093-8213-60f41c7315fb-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:41:25.107504 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.107500    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s68r\" (UniqueName: \"kubernetes.io/projected/c9e36473-b5b9-4093-8213-60f41c7315fb-kube-api-access-4s68r\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:41:25.201008 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.200977    2565 generic.go:356] "Generic (PLEG): container finished" podID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerID="73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4" exitCode=0
Sep 17 18:41:25.201154 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.201018    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"
Sep 17 18:41:25.201154 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.201063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" event={"ID":"c9e36473-b5b9-4093-8213-60f41c7315fb","Type":"ContainerDied","Data":"73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4"}
Sep 17 18:41:25.201154 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.201124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp" event={"ID":"c9e36473-b5b9-4093-8213-60f41c7315fb","Type":"ContainerDied","Data":"df519b99231c1dcd8ed9f0006d6594f998ded65d80065bd1be9567257589a0e9"}
Sep 17 18:41:25.201154 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.201152    2565 scope.go:117] "RemoveContainer" containerID="73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4"
Sep 17 18:41:25.215386 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.215368    2565 scope.go:117] "RemoveContainer" containerID="73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4"
Sep 17 18:41:25.215689 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:41:25.215669    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4\": container with ID starting with 73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4 not found: ID does not exist" containerID="73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4"
Sep 17 18:41:25.215761 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.215697    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4"} err="failed to get container status \"73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4\": rpc error: code = NotFound desc = could not find container \"73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4\": container with ID starting with 73575d530060dbf2048bd150fce2d4f9ac7a286cd8ad0bffa4ed4a0c6f39dec4 not found: ID does not exist"
Sep 17 18:41:25.224304 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.224278    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"]
Sep 17 18:41:25.227053 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.227035    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb7rhlp"]
Sep 17 18:41:25.348636 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:25.348566    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" path="/var/lib/kubelet/pods/c9e36473-b5b9-4093-8213-60f41c7315fb/volumes"
Sep 17 18:41:33.851664 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.851631    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"]
Sep 17 18:41:33.852053 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.851925    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerName="main"
Sep 17 18:41:33.852053 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.851938    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerName="main"
Sep 17 18:41:33.852053 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.851992    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9e36473-b5b9-4093-8213-60f41c7315fb" containerName="main"
Sep 17 18:41:33.856129 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.856112    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:33.859136 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.859111    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 18:41:33.859279 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.859164    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-dgjhj\""
Sep 17 18:41:33.859279 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.859215    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 18:41:33.859279 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.859229    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 18:41:33.859279 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.859247    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-424pr\""
Sep 17 18:41:33.863826 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.863802    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"]
Sep 17 18:41:33.968784 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.968752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4be21d-e370-4386-a421-52aa7a6cd069-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj\" (UID: \"ab4be21d-e370-4386-a421-52aa7a6cd069\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:33.968955 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:33.968797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfdp\" (UniqueName: \"kubernetes.io/projected/ab4be21d-e370-4386-a421-52aa7a6cd069-kube-api-access-pvfdp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj\" (UID: \"ab4be21d-e370-4386-a421-52aa7a6cd069\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:34.069146 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:34.069111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfdp\" (UniqueName: \"kubernetes.io/projected/ab4be21d-e370-4386-a421-52aa7a6cd069-kube-api-access-pvfdp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj\" (UID: \"ab4be21d-e370-4386-a421-52aa7a6cd069\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:34.069314 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:34.069194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4be21d-e370-4386-a421-52aa7a6cd069-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj\" (UID: \"ab4be21d-e370-4386-a421-52aa7a6cd069\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:34.071786 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:34.071756    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4be21d-e370-4386-a421-52aa7a6cd069-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj\" (UID: \"ab4be21d-e370-4386-a421-52aa7a6cd069\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:34.076744 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:34.076720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfdp\" (UniqueName: \"kubernetes.io/projected/ab4be21d-e370-4386-a421-52aa7a6cd069-kube-api-access-pvfdp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj\" (UID: \"ab4be21d-e370-4386-a421-52aa7a6cd069\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:34.166945 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:34.166923    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:34.435162 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:34.435079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"]
Sep 17 18:41:34.438835 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:41:34.438804    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4be21d_e370_4386_a421_52aa7a6cd069.slice/crio-b82c77e78042fb01fa1c845d1b40ce1f688e3107165bc5b73ab104d16359a9f2 WatchSource:0}: Error finding container b82c77e78042fb01fa1c845d1b40ce1f688e3107165bc5b73ab104d16359a9f2: Status 404 returned error can't find the container with id b82c77e78042fb01fa1c845d1b40ce1f688e3107165bc5b73ab104d16359a9f2
Sep 17 18:41:35.244622 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:35.244584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" event={"ID":"ab4be21d-e370-4386-a421-52aa7a6cd069","Type":"ContainerStarted","Data":"d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234"}
Sep 17 18:41:35.245037 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:35.244644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" event={"ID":"ab4be21d-e370-4386-a421-52aa7a6cd069","Type":"ContainerStarted","Data":"b82c77e78042fb01fa1c845d1b40ce1f688e3107165bc5b73ab104d16359a9f2"}
Sep 17 18:41:35.245037 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:35.244664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:41:35.263716 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:35.263675    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" podStartSLOduration=2.263661979 podStartE2EDuration="2.263661979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:35.261804408 +0000 UTC m=+3638.097573641" watchObservedRunningTime="2026-09-17 18:41:35.263661979 +0000 UTC m=+3638.099431202"
Sep 17 18:41:49.181270 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:41:49.181240    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:42:06.250548 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:06.250518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:42:07.422263 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:07.422226    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"]
Sep 17 18:42:16.250259 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:16.250170    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:22.442779 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:22.442742    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerName="main" containerID="cri-o://d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234" gracePeriod=45
Sep 17 18:42:26.250350 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:26.250313    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:36.249927 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:36.249887    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:36.250471 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:36.249988    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:42:46.250556 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:46.250517    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:52.680597 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:52.680572    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:42:52.774434 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:52.774337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4be21d-e370-4386-a421-52aa7a6cd069-tls-certs\") pod \"ab4be21d-e370-4386-a421-52aa7a6cd069\" (UID: \"ab4be21d-e370-4386-a421-52aa7a6cd069\") "
Sep 17 18:42:52.774606 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:52.774449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvfdp\" (UniqueName: \"kubernetes.io/projected/ab4be21d-e370-4386-a421-52aa7a6cd069-kube-api-access-pvfdp\") pod \"ab4be21d-e370-4386-a421-52aa7a6cd069\" (UID: \"ab4be21d-e370-4386-a421-52aa7a6cd069\") "
Sep 17 18:42:52.776679 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:52.776653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4be21d-e370-4386-a421-52aa7a6cd069-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab4be21d-e370-4386-a421-52aa7a6cd069" (UID: "ab4be21d-e370-4386-a421-52aa7a6cd069"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:52.776779 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:52.776681    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4be21d-e370-4386-a421-52aa7a6cd069-kube-api-access-pvfdp" (OuterVolumeSpecName: "kube-api-access-pvfdp") pod "ab4be21d-e370-4386-a421-52aa7a6cd069" (UID: "ab4be21d-e370-4386-a421-52aa7a6cd069"). InnerVolumeSpecName "kube-api-access-pvfdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:52.875655 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:52.875629    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvfdp\" (UniqueName: \"kubernetes.io/projected/ab4be21d-e370-4386-a421-52aa7a6cd069-kube-api-access-pvfdp\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:42:52.875655 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:52.875653    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4be21d-e370-4386-a421-52aa7a6cd069-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.537702 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.537663    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerID="d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234" exitCode=0
Sep 17 18:42:53.537875 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.537709    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"
Sep 17 18:42:53.537875 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.537712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" event={"ID":"ab4be21d-e370-4386-a421-52aa7a6cd069","Type":"ContainerDied","Data":"d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234"}
Sep 17 18:42:53.537875 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.537763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj" event={"ID":"ab4be21d-e370-4386-a421-52aa7a6cd069","Type":"ContainerDied","Data":"b82c77e78042fb01fa1c845d1b40ce1f688e3107165bc5b73ab104d16359a9f2"}
Sep 17 18:42:53.537875 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.537794    2565 scope.go:117] "RemoveContainer" containerID="d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234"
Sep 17 18:42:53.546203 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.546182    2565 scope.go:117] "RemoveContainer" containerID="d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234"
Sep 17 18:42:53.546485 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:42:53.546462    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234\": container with ID starting with d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234 not found: ID does not exist" containerID="d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234"
Sep 17 18:42:53.546534 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.546497    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234"} err="failed to get container status \"d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234\": rpc error: code = NotFound desc = could not find container \"d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234\": container with ID starting with d9bc1a6b6caab3e422b704d19a777a09e744d434ca431baf798ac30511ed1234 not found: ID does not exist"
Sep 17 18:42:53.552035 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.552012    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"]
Sep 17 18:42:53.553865 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:53.553844    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedg7bj"]
Sep 17 18:42:55.347960 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:42:55.347924    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" path="/var/lib/kubelet/pods/ab4be21d-e370-4386-a421-52aa7a6cd069/volumes"
Sep 17 18:43:09.538093 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.538059    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"]
Sep 17 18:43:09.538599 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.538346    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerName="main"
Sep 17 18:43:09.538599 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.538357    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerName="main"
Sep 17 18:43:09.538599 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.538422    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4be21d-e370-4386-a421-52aa7a6cd069" containerName="main"
Sep 17 18:43:09.541552 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.541531    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.543756 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.543734    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 18:43:09.543846 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.543734    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:43:09.555940 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.555914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"]
Sep 17 18:43:09.706504 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.706468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cae1c4-436e-46ea-aa06-165e6bdbe56d-cabundle-cert\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.706504 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.706504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-home\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.706747 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.706524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-dshm\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.706747 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.706542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kube-api-access-d5dsj\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.706747 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.706604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tmp-dir\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.706747 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.706668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tls-certs\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.706747 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.706739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-model-cache\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.706957 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.706770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kserve-provision-location\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.807842 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.807760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tls-certs\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.807842 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.807801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-model-cache\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.807842 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.807819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kserve-provision-location\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.807842 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.807844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cae1c4-436e-46ea-aa06-165e6bdbe56d-cabundle-cert\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808164 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.807865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-home\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808164 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.807893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-dshm\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808164 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.807917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kube-api-access-d5dsj\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808164 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.807948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tmp-dir\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808376 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.808270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kserve-provision-location\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808376 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.808305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-model-cache\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808586 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.808559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tmp-dir\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808697 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.808592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cae1c4-436e-46ea-aa06-165e6bdbe56d-cabundle-cert\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.808697 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.808642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-home\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.810368 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.810341    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-dshm\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.810638 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.810619    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tls-certs\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.815248 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.815225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kube-api-access-d5dsj\") pod \"prestop-hook-test-kserve-78dbc984b4-nx25v\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:09.851249 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:09.851228    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:10.125603 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:10.125512    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"]
Sep 17 18:43:10.128258 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:43:10.128227    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cae1c4_436e_46ea_aa06_165e6bdbe56d.slice/crio-41da14eb41111048d4600471b8192c4b00deb873aa558c9c97d67732a71c7ee7 WatchSource:0}: Error finding container 41da14eb41111048d4600471b8192c4b00deb873aa558c9c97d67732a71c7ee7: Status 404 returned error can't find the container with id 41da14eb41111048d4600471b8192c4b00deb873aa558c9c97d67732a71c7ee7
Sep 17 18:43:10.602343 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:10.602305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" event={"ID":"45cae1c4-436e-46ea-aa06-165e6bdbe56d","Type":"ContainerStarted","Data":"4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836"}
Sep 17 18:43:10.602753 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:10.602364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" event={"ID":"45cae1c4-436e-46ea-aa06-165e6bdbe56d","Type":"ContainerStarted","Data":"41da14eb41111048d4600471b8192c4b00deb873aa558c9c97d67732a71c7ee7"}
Sep 17 18:43:12.610930 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:12.610843    2565 generic.go:356] "Generic (PLEG): container finished" podID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerID="4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836" exitCode=0
Sep 17 18:43:12.610930 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:12.610918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" event={"ID":"45cae1c4-436e-46ea-aa06-165e6bdbe56d","Type":"ContainerDied","Data":"4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836"}
Sep 17 18:43:13.616690 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:13.616656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" event={"ID":"45cae1c4-436e-46ea-aa06-165e6bdbe56d","Type":"ContainerStarted","Data":"84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f"}
Sep 17 18:43:13.636755 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:13.636704    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" podStartSLOduration=2.560624906 podStartE2EDuration="4.636687005s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.076062099s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:13.635938025 +0000 UTC m=+3736.471707261" watchObservedRunningTime="2026-09-17 18:43:13.636687005 +0000 UTC m=+3736.472456228"
Sep 17 18:43:13.851797 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:13.851760    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:19.851717 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:19.851667    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:43:19.853228 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:19.853197    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" podUID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:43:27.822810 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:27.822772    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"]
Sep 17 18:43:27.824886 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:27.824853    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"]
Sep 17 18:43:29.181573 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:29.181529    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:43:39.181315 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:39.181273    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:43:42.844825 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:42.844721    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main" containerID="cri-o://29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89" gracePeriod=45
Sep 17 18:43:42.851663 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:42.851604    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" podUID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerName="main" containerID="cri-o://84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f" gracePeriod=45
Sep 17 18:43:49.181408 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:49.181349    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:43:49.181853 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:49.181507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:43:59.181466 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:43:59.181424    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:09.181447 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:09.181411    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:13.107947 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.107884    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:44:13.223105 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.223067    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c36295d-ff75-4328-9633-03d5678134c3-tls-certs\") pod \"0c36295d-ff75-4328-9633-03d5678134c3\" (UID: \"0c36295d-ff75-4328-9633-03d5678134c3\") "
Sep 17 18:44:13.223324 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.223129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzrrr\" (UniqueName: \"kubernetes.io/projected/0c36295d-ff75-4328-9633-03d5678134c3-kube-api-access-rzrrr\") pod \"0c36295d-ff75-4328-9633-03d5678134c3\" (UID: \"0c36295d-ff75-4328-9633-03d5678134c3\") "
Sep 17 18:44:13.225356 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.225321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c36295d-ff75-4328-9633-03d5678134c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c36295d-ff75-4328-9633-03d5678134c3" (UID: "0c36295d-ff75-4328-9633-03d5678134c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:13.225489 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.225390    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c36295d-ff75-4328-9633-03d5678134c3-kube-api-access-rzrrr" (OuterVolumeSpecName: "kube-api-access-rzrrr") pod "0c36295d-ff75-4328-9633-03d5678134c3" (UID: "0c36295d-ff75-4328-9633-03d5678134c3"). InnerVolumeSpecName "kube-api-access-rzrrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:13.324456 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.324425    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzrrr\" (UniqueName: \"kubernetes.io/projected/0c36295d-ff75-4328-9633-03d5678134c3-kube-api-access-rzrrr\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:13.324456 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.324453    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c36295d-ff75-4328-9633-03d5678134c3-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:13.846537 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.846503    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c36295d-ff75-4328-9633-03d5678134c3" containerID="29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89" exitCode=0
Sep 17 18:44:13.846746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.846544    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"
Sep 17 18:44:13.846746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.846595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" event={"ID":"0c36295d-ff75-4328-9633-03d5678134c3","Type":"ContainerDied","Data":"29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89"}
Sep 17 18:44:13.846746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.846657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd" event={"ID":"0c36295d-ff75-4328-9633-03d5678134c3","Type":"ContainerDied","Data":"56c4b5148de6a666f232c81ab95e699731da7e0c766b81c87cebd3b94f38eb4f"}
Sep 17 18:44:13.846746 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.846679    2565 scope.go:117] "RemoveContainer" containerID="29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89"
Sep 17 18:44:13.854750 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.854729    2565 scope.go:117] "RemoveContainer" containerID="29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89"
Sep 17 18:44:13.855016 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:44:13.854996    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89\": container with ID starting with 29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89 not found: ID does not exist" containerID="29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89"
Sep 17 18:44:13.855076 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.855028    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89"} err="failed to get container status \"29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89\": rpc error: code = NotFound desc = could not find container \"29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89\": container with ID starting with 29fb986d68541d0541129063169de622bb0b5f387a1445a9b7b982b7424bff89 not found: ID does not exist"
Sep 17 18:44:13.860260 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.860237    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"]
Sep 17 18:44:13.862191 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:13.862166    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7bmd"]
Sep 17 18:44:15.348341 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:15.348311    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c36295d-ff75-4328-9633-03d5678134c3" path="/var/lib/kubelet/pods/0c36295d-ff75-4328-9633-03d5678134c3/volumes"
Sep 17 18:44:28.485896 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.485873    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-78dbc984b4-nx25v_45cae1c4-436e-46ea-aa06-165e6bdbe56d/main/0.log"
Sep 17 18:44:28.488461 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.488443    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:44:28.539359 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539333    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-model-cache\") pod \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") "
Sep 17 18:44:28.539543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kube-api-access-d5dsj\") pod \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") "
Sep 17 18:44:28.539543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539406    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kserve-provision-location\") pod \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") "
Sep 17 18:44:28.539543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cae1c4-436e-46ea-aa06-165e6bdbe56d-cabundle-cert\") pod \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") "
Sep 17 18:44:28.539543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tmp-dir\") pod \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") "
Sep 17 18:44:28.539543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539492    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-home\") pod \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") "
Sep 17 18:44:28.539543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539532    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-dshm\") pod \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") "
Sep 17 18:44:28.539867 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539560    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tls-certs\") pod \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\" (UID: \"45cae1c4-436e-46ea-aa06-165e6bdbe56d\") "
Sep 17 18:44:28.539867 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539685    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-model-cache" (OuterVolumeSpecName: "model-cache") pod "45cae1c4-436e-46ea-aa06-165e6bdbe56d" (UID: "45cae1c4-436e-46ea-aa06-165e6bdbe56d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:28.539867 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539834    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-model-cache\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.540010 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cae1c4-436e-46ea-aa06-165e6bdbe56d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45cae1c4-436e-46ea-aa06-165e6bdbe56d" (UID: "45cae1c4-436e-46ea-aa06-165e6bdbe56d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:28.540078 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.539991    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45cae1c4-436e-46ea-aa06-165e6bdbe56d" (UID: "45cae1c4-436e-46ea-aa06-165e6bdbe56d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:28.540254 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.540226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-home" (OuterVolumeSpecName: "home") pod "45cae1c4-436e-46ea-aa06-165e6bdbe56d" (UID: "45cae1c4-436e-46ea-aa06-165e6bdbe56d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:28.541619 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.541592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kube-api-access-d5dsj" (OuterVolumeSpecName: "kube-api-access-d5dsj") pod "45cae1c4-436e-46ea-aa06-165e6bdbe56d" (UID: "45cae1c4-436e-46ea-aa06-165e6bdbe56d"). InnerVolumeSpecName "kube-api-access-d5dsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:28.541788 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.541771    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45cae1c4-436e-46ea-aa06-165e6bdbe56d" (UID: "45cae1c4-436e-46ea-aa06-165e6bdbe56d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:28.541835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.541778    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-dshm" (OuterVolumeSpecName: "dshm") pod "45cae1c4-436e-46ea-aa06-165e6bdbe56d" (UID: "45cae1c4-436e-46ea-aa06-165e6bdbe56d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:28.562152 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.562120    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45cae1c4-436e-46ea-aa06-165e6bdbe56d" (UID: "45cae1c4-436e-46ea-aa06-165e6bdbe56d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:28.640906 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.640834    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-home\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.640906 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.640861    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-dshm\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.640906 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.640870    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.640906 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.640880    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kube-api-access-d5dsj\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.640906 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.640889    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-kserve-provision-location\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.640906 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.640899    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cae1c4-436e-46ea-aa06-165e6bdbe56d-cabundle-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.641209 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.640928    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45cae1c4-436e-46ea-aa06-165e6bdbe56d-tmp-dir\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.908005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.907930    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-78dbc984b4-nx25v_45cae1c4-436e-46ea-aa06-165e6bdbe56d/main/0.log"
Sep 17 18:44:28.908324 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.908297    2565 generic.go:356] "Generic (PLEG): container finished" podID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerID="84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f" exitCode=137
Sep 17 18:44:28.908434 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.908352    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"
Sep 17 18:44:28.908434 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.908350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" event={"ID":"45cae1c4-436e-46ea-aa06-165e6bdbe56d","Type":"ContainerDied","Data":"84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f"}
Sep 17 18:44:28.908558 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.908442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v" event={"ID":"45cae1c4-436e-46ea-aa06-165e6bdbe56d","Type":"ContainerDied","Data":"41da14eb41111048d4600471b8192c4b00deb873aa558c9c97d67732a71c7ee7"}
Sep 17 18:44:28.908558 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.908466    2565 scope.go:117] "RemoveContainer" containerID="84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f"
Sep 17 18:44:28.916867 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.916850    2565 scope.go:117] "RemoveContainer" containerID="4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836"
Sep 17 18:44:28.925452 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.925432    2565 scope.go:117] "RemoveContainer" containerID="84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f"
Sep 17 18:44:28.925735 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:44:28.925716    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f\": container with ID starting with 84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f not found: ID does not exist" containerID="84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f"
Sep 17 18:44:28.925804 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.925743    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f"} err="failed to get container status \"84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f\": rpc error: code = NotFound desc = could not find container \"84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f\": container with ID starting with 84ef5459e1a4289521387d719a26d0cedc1b1435be528712631b04f244898c3f not found: ID does not exist"
Sep 17 18:44:28.925804 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.925760    2565 scope.go:117] "RemoveContainer" containerID="4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836"
Sep 17 18:44:28.926019 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:44:28.926000    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836\": container with ID starting with 4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836 not found: ID does not exist" containerID="4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836"
Sep 17 18:44:28.926076 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.926028    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836"} err="failed to get container status \"4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836\": rpc error: code = NotFound desc = could not find container \"4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836\": container with ID starting with 4e524bd82d861d1c7e8faf9d60f0c34bc65b800db64507e217a1f7157bc02836 not found: ID does not exist"
Sep 17 18:44:28.929135 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.929110    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"]
Sep 17 18:44:28.932680 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:28.932660    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-nx25v"]
Sep 17 18:44:29.348816 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:29.348785    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" path="/var/lib/kubelet/pods/45cae1c4-436e-46ea-aa06-165e6bdbe56d/volumes"
Sep 17 18:44:49.590622 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.590582    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"]
Sep 17 18:44:49.591150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.590953    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerName="storage-initializer"
Sep 17 18:44:49.591150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.590972    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerName="storage-initializer"
Sep 17 18:44:49.591150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.590990    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerName="main"
Sep 17 18:44:49.591150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.590999    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerName="main"
Sep 17 18:44:49.591150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.591012    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main"
Sep 17 18:44:49.591150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.591021    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main"
Sep 17 18:44:49.591150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.591099    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c36295d-ff75-4328-9633-03d5678134c3" containerName="main"
Sep 17 18:44:49.591150 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.591109    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45cae1c4-436e-46ea-aa06-165e6bdbe56d" containerName="main"
Sep 17 18:44:49.594056 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.594034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:49.596087 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.596064    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 18:44:49.596732 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.596714    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 18:44:49.596840 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.596764    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 18:44:49.597446 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.597259    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-xhz7f\""
Sep 17 18:44:49.597446 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.597379    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-kglws\""
Sep 17 18:44:49.603572 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.603549    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"]
Sep 17 18:44:49.705005 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.704973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42539da-8af7-4f80-90e4-b912704a8957-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7\" (UID: \"b42539da-8af7-4f80-90e4-b912704a8957\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:49.705149 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.705013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sd8w\" (UniqueName: \"kubernetes.io/projected/b42539da-8af7-4f80-90e4-b912704a8957-kube-api-access-8sd8w\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7\" (UID: \"b42539da-8af7-4f80-90e4-b912704a8957\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:49.806038 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.806003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42539da-8af7-4f80-90e4-b912704a8957-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7\" (UID: \"b42539da-8af7-4f80-90e4-b912704a8957\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:49.806196 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.806047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sd8w\" (UniqueName: \"kubernetes.io/projected/b42539da-8af7-4f80-90e4-b912704a8957-kube-api-access-8sd8w\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7\" (UID: \"b42539da-8af7-4f80-90e4-b912704a8957\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:49.808594 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.808574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42539da-8af7-4f80-90e4-b912704a8957-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7\" (UID: \"b42539da-8af7-4f80-90e4-b912704a8957\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:49.813199 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.813175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sd8w\" (UniqueName: \"kubernetes.io/projected/b42539da-8af7-4f80-90e4-b912704a8957-kube-api-access-8sd8w\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7\" (UID: \"b42539da-8af7-4f80-90e4-b912704a8957\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:49.909129 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:49.909093    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:50.208692 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:50.208608    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"]
Sep 17 18:44:50.211377 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:44:50.211350    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42539da_8af7_4f80_90e4_b912704a8957.slice/crio-95b75c383322b102b9de09dd992a25cdea6422aa2af30fd1cc45050e9c4dab68 WatchSource:0}: Error finding container 95b75c383322b102b9de09dd992a25cdea6422aa2af30fd1cc45050e9c4dab68: Status 404 returned error can't find the container with id 95b75c383322b102b9de09dd992a25cdea6422aa2af30fd1cc45050e9c4dab68
Sep 17 18:44:50.992183 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:50.992133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" event={"ID":"b42539da-8af7-4f80-90e4-b912704a8957","Type":"ContainerStarted","Data":"1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a"}
Sep 17 18:44:50.992611 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:50.992194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" event={"ID":"b42539da-8af7-4f80-90e4-b912704a8957","Type":"ContainerStarted","Data":"95b75c383322b102b9de09dd992a25cdea6422aa2af30fd1cc45050e9c4dab68"}
Sep 17 18:44:50.992611 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:50.992237    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:44:51.009520 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:44:51.009458    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" podStartSLOduration=2.009439267 podStartE2EDuration="2.009439267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:51.007595392 +0000 UTC m=+3833.843364615" watchObservedRunningTime="2026-09-17 18:44:51.009439267 +0000 UTC m=+3833.845208493"
Sep 17 18:45:21.997651 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:45:21.997615    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:45:32.310793 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:45:32.310758    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"]
Sep 17 18:45:41.997926 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:45:41.997889    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" podUID="b42539da-8af7-4f80-90e4-b912704a8957" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:47.330266 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:45:47.330202    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" podUID="b42539da-8af7-4f80-90e4-b912704a8957" containerName="main" containerID="cri-o://1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a" gracePeriod=45
Sep 17 18:45:51.998418 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:45:51.998360    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" podUID="b42539da-8af7-4f80-90e4-b912704a8957" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:59.541976 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:45:59.541945    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:45:59.545884 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:45:59.545860    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jk425_ef7c1c2f-a805-434a-9537-bed7df1e3570/ovn-acl-logging/0.log"
Sep 17 18:46:01.997759 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:01.997722    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" podUID="b42539da-8af7-4f80-90e4-b912704a8957" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:46:01.998144 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:01.997834    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:46:11.998003 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:11.997961    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" podUID="b42539da-8af7-4f80-90e4-b912704a8957" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:46:17.573361 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:17.573332    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:46:17.691995 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:17.691958    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sd8w\" (UniqueName: \"kubernetes.io/projected/b42539da-8af7-4f80-90e4-b912704a8957-kube-api-access-8sd8w\") pod \"b42539da-8af7-4f80-90e4-b912704a8957\" (UID: \"b42539da-8af7-4f80-90e4-b912704a8957\") "
Sep 17 18:46:17.692163 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:17.692051    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42539da-8af7-4f80-90e4-b912704a8957-tls-certs\") pod \"b42539da-8af7-4f80-90e4-b912704a8957\" (UID: \"b42539da-8af7-4f80-90e4-b912704a8957\") "
Sep 17 18:46:17.694225 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:17.694195    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42539da-8af7-4f80-90e4-b912704a8957-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b42539da-8af7-4f80-90e4-b912704a8957" (UID: "b42539da-8af7-4f80-90e4-b912704a8957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:17.694343 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:17.694298    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42539da-8af7-4f80-90e4-b912704a8957-kube-api-access-8sd8w" (OuterVolumeSpecName: "kube-api-access-8sd8w") pod "b42539da-8af7-4f80-90e4-b912704a8957" (UID: "b42539da-8af7-4f80-90e4-b912704a8957"). InnerVolumeSpecName "kube-api-access-8sd8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:17.792883 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:17.792834    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42539da-8af7-4f80-90e4-b912704a8957-tls-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:46:17.792883 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:17.792878    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sd8w\" (UniqueName: \"kubernetes.io/projected/b42539da-8af7-4f80-90e4-b912704a8957-kube-api-access-8sd8w\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Sep 17 18:46:18.301211 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.301172    2565 generic.go:356] "Generic (PLEG): container finished" podID="b42539da-8af7-4f80-90e4-b912704a8957" containerID="1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a" exitCode=0
Sep 17 18:46:18.301412 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.301216    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"
Sep 17 18:46:18.301412 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.301258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" event={"ID":"b42539da-8af7-4f80-90e4-b912704a8957","Type":"ContainerDied","Data":"1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a"}
Sep 17 18:46:18.301412 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.301309    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7" event={"ID":"b42539da-8af7-4f80-90e4-b912704a8957","Type":"ContainerDied","Data":"95b75c383322b102b9de09dd992a25cdea6422aa2af30fd1cc45050e9c4dab68"}
Sep 17 18:46:18.301412 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.301331    2565 scope.go:117] "RemoveContainer" containerID="1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a"
Sep 17 18:46:18.309841 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.309823    2565 scope.go:117] "RemoveContainer" containerID="1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a"
Sep 17 18:46:18.310107 ip-10-0-129-186 kubenswrapper[2565]: E0917 18:46:18.310090    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a\": container with ID starting with 1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a not found: ID does not exist" containerID="1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a"
Sep 17 18:46:18.310155 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.310114    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a"} err="failed to get container status \"1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a\": rpc error: code = NotFound desc = could not find container \"1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a\": container with ID starting with 1cd42472c17f61e763a9a891a09fd7fb64dbc5a9be484d948984c8cdeba8612a not found: ID does not exist"
Sep 17 18:46:18.320611 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.320588    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"]
Sep 17 18:46:18.322816 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:18.322794    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-9wws7"]
Sep 17 18:46:19.347631 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:19.347597    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b42539da-8af7-4f80-90e4-b912704a8957" path="/var/lib/kubelet/pods/b42539da-8af7-4f80-90e4-b912704a8957/volumes"
Sep 17 18:46:46.450329 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:46.450211    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/0.log"
Sep 17 18:46:46.454980 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:46.454953    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/1.log"
Sep 17 18:46:47.260181 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:47.260148    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/0.log"
Sep 17 18:46:47.265260 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:47.265237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/1.log"
Sep 17 18:46:48.351950 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:48.351906    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-czwqb_61af4be2-2835-47be-8d79-7e7cd3a26832/authorino/0.log"
Sep 17 18:46:48.395440 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:48.395416    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-g6gv9_b709f81d-3c1a-4fe2-bdac-364bbab10227/kuadrant-console-plugin/0.log"
Sep 17 18:46:57.482023 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:57.481991    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v47zx_45898ed0-e311-492a-ab02-7851051c85a9/global-pull-secret-syncer/0.log"
Sep 17 18:46:57.655576 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:57.655545    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s7zjm_0868d5ee-d875-45d7-866e-5678b9f1411d/konnectivity-agent/0.log"
Sep 17 18:46:57.679025 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:46:57.679000    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-186.ec2.internal_81daa90f8e3c12260694003f97da072d/haproxy/0.log"
Sep 17 18:47:02.352300 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:02.352263    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-czwqb_61af4be2-2835-47be-8d79-7e7cd3a26832/authorino/0.log"
Sep 17 18:47:02.469222 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:02.469194    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-g6gv9_b709f81d-3c1a-4fe2-bdac-364bbab10227/kuadrant-console-plugin/0.log"
Sep 17 18:47:04.835386 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:04.835352    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf6cw_8acb1be4-db48-44cc-883c-1ed598a3cd87/node-exporter/0.log"
Sep 17 18:47:04.859528 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:04.859499    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf6cw_8acb1be4-db48-44cc-883c-1ed598a3cd87/kube-rbac-proxy/0.log"
Sep 17 18:47:04.882472 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:04.882439    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf6cw_8acb1be4-db48-44cc-883c-1ed598a3cd87/init-textfile/0.log"
Sep 17 18:47:08.548910 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.548874    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"]
Sep 17 18:47:08.549291 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.549130    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42539da-8af7-4f80-90e4-b912704a8957" containerName="main"
Sep 17 18:47:08.549291 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.549141    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42539da-8af7-4f80-90e4-b912704a8957" containerName="main"
Sep 17 18:47:08.549291 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.549190    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42539da-8af7-4f80-90e4-b912704a8957" containerName="main"
Sep 17 18:47:08.551867 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.551845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.553970 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.553950    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rx6xt\"/\"kube-root-ca.crt\""
Sep 17 18:47:08.554561 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.554546    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rx6xt\"/\"default-dockercfg-lm5ng\""
Sep 17 18:47:08.554608 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.554584    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rx6xt\"/\"openshift-service-ca.crt\""
Sep 17 18:47:08.563286 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.563265    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"]
Sep 17 18:47:08.665992 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.665948    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-podres\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.665992 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.665993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-proc\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.666242 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.666066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-sys\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.666242 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.666088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz86n\" (UniqueName: \"kubernetes.io/projected/29bbb0f8-e5b9-48d6-b165-547bb7320b63-kube-api-access-dz86n\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.666242 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.666118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-lib-modules\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766491 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-lib-modules\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-podres\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-proc\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-lib-modules\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-sys\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz86n\" (UniqueName: \"kubernetes.io/projected/29bbb0f8-e5b9-48d6-b165-547bb7320b63-kube-api-access-dz86n\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-proc\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-podres\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.766701 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.766684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29bbb0f8-e5b9-48d6-b165-547bb7320b63-sys\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.774264 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.774238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz86n\" (UniqueName: \"kubernetes.io/projected/29bbb0f8-e5b9-48d6-b165-547bb7320b63-kube-api-access-dz86n\") pod \"perf-node-gather-daemonset-h4jfq\" (UID: \"29bbb0f8-e5b9-48d6-b165-547bb7320b63\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:08.862046 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:08.861963    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:09.129142 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:09.128943    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"]
Sep 17 18:47:09.131833 ip-10-0-129-186 kubenswrapper[2565]: W0917 18:47:09.131805    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod29bbb0f8_e5b9_48d6_b165_547bb7320b63.slice/crio-635bd5a2ae71df7472e98d1319425137d016527e03b52b7246090e24f20b1adb WatchSource:0}: Error finding container 635bd5a2ae71df7472e98d1319425137d016527e03b52b7246090e24f20b1adb: Status 404 returned error can't find the container with id 635bd5a2ae71df7472e98d1319425137d016527e03b52b7246090e24f20b1adb
Sep 17 18:47:09.476542 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:09.476509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq" event={"ID":"29bbb0f8-e5b9-48d6-b165-547bb7320b63","Type":"ContainerStarted","Data":"7329e807e7fe799ef413dab54735f299fabefa90f822d51dfb8469192b0a9145"}
Sep 17 18:47:09.476716 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:09.476559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq" event={"ID":"29bbb0f8-e5b9-48d6-b165-547bb7320b63","Type":"ContainerStarted","Data":"635bd5a2ae71df7472e98d1319425137d016527e03b52b7246090e24f20b1adb"}
Sep 17 18:47:09.476716 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:09.476642    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:09.491821 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:09.491778    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq" podStartSLOduration=1.491767386 podStartE2EDuration="1.491767386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:09.4911082 +0000 UTC m=+3972.326877424" watchObservedRunningTime="2026-09-17 18:47:09.491767386 +0000 UTC m=+3972.327536608"
Sep 17 18:47:09.925377 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:09.925349    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6nx7f_96fa9c3b-e4e8-446d-984b-b78f26e4b8b3/dns/0.log"
Sep 17 18:47:09.945709 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:09.945687    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6nx7f_96fa9c3b-e4e8-446d-984b-b78f26e4b8b3/kube-rbac-proxy/0.log"
Sep 17 18:47:10.081521 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:10.081488    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bvd5m_1ce68719-e9b2-4fee-b1d2-562bd57bca45/dns-node-resolver/0.log"
Sep 17 18:47:10.636543 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:10.636504    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-794f84759f-p2xmg_2c51397f-ff44-4b47-9a0e-2e9df383c07b/registry/0.log"
Sep 17 18:47:10.666226 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:10.666195    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2vkjk_fab829e2-cb5a-4779-bff8-0a6ae8a78c3f/node-ca/0.log"
Sep 17 18:47:11.778330 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:11.778299    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/0.log"
Sep 17 18:47:11.783230 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:11.783189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/1.log"
Sep 17 18:47:12.449476 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:12.449444    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-snzb8_93732b61-bff8-483a-bf62-a0a298041e71/serve-healthcheck-canary/0.log"
Sep 17 18:47:13.120693 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:13.120663    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fj7nq_fbc275f3-5fe6-40d8-b267-5cc0c7e7e143/kube-rbac-proxy/0.log"
Sep 17 18:47:13.141685 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:13.141658    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fj7nq_fbc275f3-5fe6-40d8-b267-5cc0c7e7e143/exporter/0.log"
Sep 17 18:47:13.163019 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:13.162996    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fj7nq_fbc275f3-5fe6-40d8-b267-5cc0c7e7e143/extractor/0.log"
Sep 17 18:47:15.489277 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:15.489252    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-h4jfq"
Sep 17 18:47:16.259502 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:16.259471    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68dc9b9bc-s7hzh_33497543-6e0c-4b2c-96e6-6690af1c6a82/manager/0.log"
Sep 17 18:47:18.659835 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:18.659791    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/0.log"
Sep 17 18:47:18.664574 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:18.664547    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/1.log"
Sep 17 18:47:19.936427 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:19.936360    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/0.log"
Sep 17 18:47:19.941493 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:19.941453    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-rk2jc_fc9ca3db-8b65-44b4-a00e-71ff7578fefb/istio-proxy/1.log"
Sep 17 18:47:23.808489 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:23.808440    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-h4748_01a47f0c-e831-4b54-b4c8-6171316436bb/migrator/0.log"
Sep 17 18:47:23.829285 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:23.829255    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-h4748_01a47f0c-e831-4b54-b4c8-6171316436bb/graceful-termination/0.log"
Sep 17 18:47:25.621370 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:25.621342    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvpl4_c3415d80-dec7-47e2-aded-58c3bd05537f/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:47:25.642858 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:25.642832    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvpl4_c3415d80-dec7-47e2-aded-58c3bd05537f/egress-router-binary-copy/0.log"
Sep 17 18:47:25.667503 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:25.667480    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvpl4_c3415d80-dec7-47e2-aded-58c3bd05537f/cni-plugins/0.log"
Sep 17 18:47:25.687699 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:25.687678    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvpl4_c3415d80-dec7-47e2-aded-58c3bd05537f/bond-cni-plugin/0.log"
Sep 17 18:47:25.708212 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:25.708177    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvpl4_c3415d80-dec7-47e2-aded-58c3bd05537f/routeoverride-cni/0.log"
Sep 17 18:47:25.728776 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:25.728754    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvpl4_c3415d80-dec7-47e2-aded-58c3bd05537f/whereabouts-cni-bincopy/0.log"
Sep 17 18:47:25.749678 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:25.749654    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvpl4_c3415d80-dec7-47e2-aded-58c3bd05537f/whereabouts-cni/0.log"
Sep 17 18:47:25.827219 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:25.827184    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lpzgv_94f278e6-8be6-42ba-b87a-be298b200738/kube-multus/0.log"
Sep 17 18:47:26.002038 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:26.002009    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x9dql_06e9329e-dfb9-4ca8-9807-995754f7f505/network-metrics-daemon/0.log"
Sep 17 18:47:26.023035 ip-10-0-129-186 kubenswrapper[2565]: I0917 18:47:26.023010    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x9dql_06e9329e-dfb9-4ca8-9807-995754f7f505/kube-rbac-proxy/0.log"