Sep 16 19:39:12.039262 ip-10-0-136-120 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 19:39:12.412942 ip-10-0-136-120 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:12.412942 ip-10-0-136-120 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 19:39:12.412942 ip-10-0-136-120 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:12.412942 ip-10-0-136-120 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 19:39:12.412942 ip-10-0-136-120 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:12.414148 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.413914    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 19:39:12.418446 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418431    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:12.418446 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418446    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418451    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418455    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418459    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418462    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418465    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418468    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418478    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418481    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418485    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418487    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418490    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418493    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418495    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418498    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418501    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418503    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418506    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418509    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418511    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:12.418514 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418514    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418517    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418522    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418526    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418529    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418532    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418535    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418540    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418543    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418546    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418549    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418552    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418556    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418559    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418561    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418564    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418567    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418569    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418573    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:12.419035 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418575    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418578    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418580    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418583    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418587    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418590    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418593    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418595    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418598    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418601    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418604    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418607    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418609    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418611    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418614    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418617    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418620    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418623    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418626    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:12.419614 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418628    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418631    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418633    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418636    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418638    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418641    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418644    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418646    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418651    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418653    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418656    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418658    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418660    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418664    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418666    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418668    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418671    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418673    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418676    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418679    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:12.420189 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418681    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418683    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418686    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418688    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418691    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418693    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418696    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418699    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418702    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418704    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418707    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418709    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418712    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418714    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418717    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418720    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418722    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418725    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418727    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:12.420953 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.418730    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:12.422165 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422137    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:12.422165 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422154    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:12.422165 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422160    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:12.422165 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422165    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:12.422165 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422169    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:12.422165 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422174    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422180    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422185    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422190    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422194    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422199    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422207    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422212    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422216    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422221    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422225    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422229    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422233    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422237    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422240    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422244    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422248    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422253    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422257    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422264    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:12.422519 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422268    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422272    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422276    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422280    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422284    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422289    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422293    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422297    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422302    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422306    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422310    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422320    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422325    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422330    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422336    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422341    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422345    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422350    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422354    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422358    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:12.423133 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422363    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422373    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422379    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422389    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422394    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422399    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422403    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422407    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422412    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422419    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422425    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422430    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422434    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422439    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422444    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422453    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422458    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422463    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422467    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:12.423654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422472    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422476    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422481    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422485    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422490    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422494    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422499    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422504    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422511    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422523    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422532    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422538    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422543    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422548    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422552    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422557    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422561    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422566    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422570    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:12.424150 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422575    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422584    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422588    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422686    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422815    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422822    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422827    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422831    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422836    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422842    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422846    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422851    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422855    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422860    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422865    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.422869    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423295    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423302    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423307    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:12.424619 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423312    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423315    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423317    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423321    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423324    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423326    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423329    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423332    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423336    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423340    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423344    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423347    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423351    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423355    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423358    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423361    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423364    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423367    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423370    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:12.425105 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423373    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423376    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423379    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423382    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423386    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423388    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423391    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423394    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423396    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423399    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423402    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423404    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423407    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423410    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423412    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423415    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423417    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423420    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423422    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423425    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:12.425575 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423427    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423429    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423432    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423435    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423438    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423443    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423446    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423449    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423452    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423454    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423457    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423459    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423462    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423465    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423467    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423470    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423472    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423475    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423477    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423481    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:12.426088 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423484    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423486    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423489    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423492    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423495    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423498    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423500    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423503    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423505    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423508    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423511    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423513    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423516    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423518    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423521    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423523    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423526    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423530    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423532    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423535    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:12.426580 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423537    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423540    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423542    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423545    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423547    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423550    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423553    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423556    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423559    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423561    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423564    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423567    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423569    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423572    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423574    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423577    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.423579    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423653    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423701    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423709    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423714    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 19:39:12.427078 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423719    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423722    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423726    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423731    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423734    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423737    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423741    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423745    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423749    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423752    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423755    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423758    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423761    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423764    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423769    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423771    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423775    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423778    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423782    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423786    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423789    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423792    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423795    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423798    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 19:39:12.427577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423801    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423804    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423807    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423810    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423814    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423817    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423820    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423823    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423827    2561 flags.go:64] FLAG: --enable-server="true"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423830    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423834    2561 flags.go:64] FLAG: --event-burst="100"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423838    2561 flags.go:64] FLAG: --event-qps="50"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423840    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423844    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423847    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423850    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423856    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423859    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423862    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423865    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423868    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423872    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423875    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423878    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423881    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 19:39:12.428209 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423883    2561 flags.go:64] FLAG: --feature-gates=""
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423887    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423891    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423894    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423911    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423916    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423919    2561 flags.go:64] FLAG: --help="false"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423922    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423925    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423928    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423931    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423934    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423938    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423941    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423944    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423946    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423949    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423952    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423956    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423959    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423963    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423966    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423969    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423974    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 19:39:12.428893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423977    2561 flags.go:64] FLAG: --lock-file=""
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423979    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423982    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423985    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423991    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423995    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.423998    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424000    2561 flags.go:64] FLAG: --logging-format="text"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424003    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424007    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424010    2561 flags.go:64] FLAG: --manifest-url=""
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424012    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424020    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424023    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424027    2561 flags.go:64] FLAG: --max-pods="110"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424031    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424036    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424040    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424045    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424050    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424053    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424056    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424062    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424066    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424069    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 19:39:12.429503 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424072    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424074    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424080    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424085    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424089    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424092    2561 flags.go:64] FLAG: --port="10250"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424095    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424100    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-008c69f83e7a29484"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424103    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424106    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424109    2561 flags.go:64] FLAG: --register-node="true"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424112    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424116    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424119    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424122    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424125    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424129    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424132    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424135    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424137    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424140    2561 flags.go:64] FLAG: --runonce="false"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424143    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424146    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424149    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424152    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 19:39:12.430132 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424155    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424158    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424162    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424165    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424168    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424171    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424174    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424177    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424180    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424183    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424187    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424193    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424196    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424198    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424203    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424206    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424209    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424212    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424215    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424218    2561 flags.go:64] FLAG: --v="2"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424222    2561 flags.go:64] FLAG: --version="false"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424226    2561 flags.go:64] FLAG: --vmodule=""
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424230    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.424233    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424347    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:12.430749 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424350    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424353    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424355    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424358    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424361    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424363    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424366    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424368    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424371    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424374    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424377    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424379    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424382    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424385    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424387    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424390    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424392    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424396    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424399    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424402    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:12.431416 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424405    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424408    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424411    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424414    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424417    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424420    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424423    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424426    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424428    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424434    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424437    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424441    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424445    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424448    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424450    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424453    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424455    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424458    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424460    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:12.431926 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424463    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424466    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424468    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424471    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424473    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424477    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424481    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424485    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424488    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424492    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424496    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424500    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424503    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424506    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424510    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424512    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424515    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424518    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424520    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:12.432399 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424523    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424526    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424528    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424531    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424534    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424536    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424539    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424541    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424544    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424546    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424549    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424551    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424554    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424556    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424559    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424562    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424564    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424567    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424569    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424572    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:12.432936 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424574    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424577    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424579    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424583    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424586    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424588    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424591    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424595    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424597    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424600    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424603    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424606    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424608    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424611    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424614    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424616    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424619    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424622    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424625    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:12.433431 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.424627    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.425221    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.432122    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.432137    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.432869    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.432875    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432946    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432952    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432955    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432959    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432962    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432965    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432968    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432971    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432974    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:12.433913 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432977    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432979    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432982    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432984    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432988    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432991    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432993    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432996    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.432999    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433002    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433006    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433009    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433013    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433016    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433018    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433021    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433024    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433028    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433031    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:12.434304 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433041    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433046    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433049    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433052    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433055    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433057    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433060    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433062    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433065    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433068    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433071    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433073    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433076    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433079    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433081    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433084    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433086    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433090    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433092    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433095    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:12.434768 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433097    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433100    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433103    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433105    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433108    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433110    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433113    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433116    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433118    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433122    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433124    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433127    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433129    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433132    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433136    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433139    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433142    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433144    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433147    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433150    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:12.435275 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433153    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433155    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433158    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433160    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433163    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433165    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433168    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433170    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433173    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433176    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433178    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433181    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433183    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433187    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433191    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433193    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433196    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433199    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433202    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:12.435782 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433205    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433208    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433210    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433213    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433216    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433218    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433221    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433224    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433227    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433229    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433232    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433235    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.433240    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433369    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433375    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:12.436302 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433378    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433382    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433385    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433388    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433391    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433393    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433396    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433399    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433402    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433405    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433407    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433410    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433412    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433415    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433417    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433421    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433426    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433429    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433431    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433434    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:12.436681 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433437    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433439    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433442    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433445    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433448    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433451    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433454    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433456    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433459    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433462    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433464    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433467    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433469    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433473    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433477    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433480    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433483    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433486    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433489    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:12.437192 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433491    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433494    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433496    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433499    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433502    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433505    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433507    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433510    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433513    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433516    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433518    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433521    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433523    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433526    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433529    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433532    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433534    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433537    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433539    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433542    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:12.437666 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433546    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433548    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433551    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433553    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433555    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433558    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433561    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433563    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433566    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433568    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433571    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433574    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433577    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433579    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433582    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433584    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433587    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433590    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433593    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433595    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:12.438178 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433598    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433600    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433603    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433605    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433609    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433612    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433616    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433619    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433622    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433624    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433627    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433630    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433633    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433636    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433638    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433641    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433643    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:12.438669 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:12.433646    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:12.439130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.433650    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:12.439130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.433756    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 19:39:12.439130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.435645    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 19:39:12.439130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.436643    2561 server.go:1019] "Starting client certificate rotation"
Sep 16 19:39:12.439130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.436742    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 19:39:12.439130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.436785    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 19:39:12.439130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.438973    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 19:39:12.440096 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.440080    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 19:39:12.455921 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.455890    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 16 19:39:12.459807 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.459788    2561 log.go:25] "Validated CRI v1 image API"
Sep 16 19:39:12.460694 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.460662    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 19:39:12.463245 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.463227    2561 fs.go:135] Filesystem UUIDs: map[2ca642cf-6a5f-4932-ba55-9c5e7eafd942:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fb9e1ee1-1174-423f-9d6f-85c667f18076:/dev/nvme0n1p4]
Sep 16 19:39:12.463322 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.463245    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 19:39:12.467449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.467341    2561 manager.go:216] Machine: {Timestamp:2026-09-16 19:39:12.466862227 +0000 UTC m=+0.322208744 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099864 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec200e501d00a44e0625cd0ccae15a77 SystemUUID:ec200e50-1d00-a44e-0625-cd0ccae15a77 BootID:5dd22ce3-1fe5-4663-80f2-aab0c7e8bd25 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:70:82:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:70:82:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:5d:45:e2:ba:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 19:39:12.467449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.467445    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 19:39:12.467566 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.467519    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 19:39:12.470326 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.470304    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 19:39:12.470767 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.470740    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 19:39:12.470924 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.470770    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 19:39:12.471934 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.471924    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 19:39:12.471974 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.471937    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 19:39:12.471974 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.471944    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 19:39:12.472464 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.472454    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 19:39:12.473576 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.473564    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 19:39:12.477130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477118    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 19:39:12.477249 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477239    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 19:39:12.477281 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477272    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 19:39:12.477322 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477291    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 19:39:12.477322 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477303    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 19:39:12.477322 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477315    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 16 19:39:12.477437 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477373    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 19:39:12.477781 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477765    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 19:39:12.477846 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.477791    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 19:39:12.480091 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.480076    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 19:39:12.482227 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.482214    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 19:39:12.482285 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.482244    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 19:39:12.483678 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483667    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483683    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483689    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483695    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483703    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483708    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483721    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483726    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483733    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 19:39:12.483736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483739    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 19:39:12.483982 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483748    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 19:39:12.483982 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.483753    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 19:39:12.484517 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.484507    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 19:39:12.484517 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.484517    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 19:39:12.488686 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.488670    2561 server.go:1268] "Started kubelet"
Sep 16 19:39:12.488813 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.488766    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 19:39:12.488854 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.488835    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 19:39:12.488927 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.488787    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 19:39:12.489616 ip-10-0-136-120 systemd[1]: Started Kubernetes Kubelet.
Sep 16 19:39:12.489967 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.489954    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 19:39:12.490247 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.490229    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 19:39:12.495334 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.495301    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 19:39:12.495483 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.495431    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 19:39:12.495483 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.495431    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 19:39:12.495590 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.495440    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 19:39:12.499163 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.498168    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-120.ec2.internal.18d5e465b88bd3a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-120.ec2.internal,UID:ip-10-0-136-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-120.ec2.internal,},FirstTimestamp:2026-09-16 19:39:12.488641443 +0000 UTC m=+0.343987961,LastTimestamp:2026-09-16 19:39:12.488641443 +0000 UTC m=+0.343987961,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-120.ec2.internal,}"
Sep 16 19:39:12.500337 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.500318    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 19:39:12.500807 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.500788    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 19:39:12.500978 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.500958    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:12.501442 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.501423    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 19:39:12.501442 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.501426    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 19:39:12.501564 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.501454    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 19:39:12.501564 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.501542    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 19:39:12.501564 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.501550    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 19:39:12.502435 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.502423    2561 factory.go:153] Registering CRI-O factory
Sep 16 19:39:12.502494 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.502437    2561 factory.go:223] Registration of the crio container factory successfully
Sep 16 19:39:12.502534 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.502492    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 19:39:12.502534 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.502509    2561 factory.go:55] Registering systemd factory
Sep 16 19:39:12.502534 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.502517    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 16 19:39:12.502534 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.502534    2561 factory.go:103] Registering Raw factory
Sep 16 19:39:12.502633 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.502548    2561 manager.go:1196] Started watching for new ooms in manager
Sep 16 19:39:12.502701 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.502679    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 19:39:12.502750 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.502732    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 19:39:12.503028 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.503005    2561 manager.go:349] Starting recovery of all containers
Sep 16 19:39:12.511679 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.511657    2561 manager.go:354] Recovery completed
Sep 16 19:39:12.517679 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.517652    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:12.519739 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.519658    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:12.519739 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.519710    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:12.519739 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.519727    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:12.520523 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.520508    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 19:39:12.520523 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.520520    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:12.520650 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.520522    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 19:39:12.520650 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.520565    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 19:39:12.521249 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.521234    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:12.521319 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.521265    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:12.521319 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.521277    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:12.522628 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.522541    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-120.ec2.internal.18d5e465ba658fec  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-120.ec2.internal,UID:ip-10-0-136-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-120.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-120.ec2.internal,},FirstTimestamp:2026-09-16 19:39:12.519688172 +0000 UTC m=+0.375034697,LastTimestamp:2026-09-16 19:39:12.519688172 +0000 UTC m=+0.375034697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-120.ec2.internal,}"
Sep 16 19:39:12.522815 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.522796    2561 policy_none.go:49] "None policy: Start"
Sep 16 19:39:12.522815 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.522815    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 19:39:12.522974 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.522826    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 19:39:12.524940 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.524927    2561 policy_none.go:47] "Start"
Sep 16 19:39:12.530528 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.530506    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jv86v"
Sep 16 19:39:12.534607 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.533056    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-120.ec2.internal.18d5e465ba660e0e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-120.ec2.internal,UID:ip-10-0-136-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-120.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-120.ec2.internal,},FirstTimestamp:2026-09-16 19:39:12.519720462 +0000 UTC m=+0.375066986,LastTimestamp:2026-09-16 19:39:12.519720462 +0000 UTC m=+0.375066986,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-120.ec2.internal,}"
Sep 16 19:39:12.535815 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.535740    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-120.ec2.internal.18d5e465ba6638c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-120.ec2.internal,UID:ip-10-0-136-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-120.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-120.ec2.internal,},FirstTimestamp:2026-09-16 19:39:12.5197314 +0000 UTC m=+0.375077916,LastTimestamp:2026-09-16 19:39:12.5197314 +0000 UTC m=+0.375077916,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-120.ec2.internal,}"
Sep 16 19:39:12.538920 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.538882    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jv86v"
Sep 16 19:39:12.548645 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.548630    2561 manager.go:337] "Starting Device Plugin manager"
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.548674    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.548688    2561 server.go:87] "Starting device plugin registration server"
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.548971    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.548985    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.549074    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.549147    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.549154    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.549661    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 19:39:12.572828 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.549714    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:12.592158 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.592115    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 19:39:12.593302 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.593274    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 19:39:12.593405 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.593330    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 19:39:12.593405 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.593356    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 19:39:12.593405 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.593398    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 19:39:12.595767 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.595743    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:12.649833 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.649802    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:12.650925 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.650891    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:12.651042 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.650940    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:12.651042 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.650956    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:12.651042 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.650986    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.660626 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.660603    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.660730 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.660630    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-120.ec2.internal\": node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:12.679141 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.679061    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:12.694025 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.693995    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal"]
Sep 16 19:39:12.694111 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.694066    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:12.695052 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.695030    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:12.695150 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.695063    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:12.695150 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.695078    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:12.697451 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.697436    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:12.697579 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.697552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.697629 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.697594    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:12.698176 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.698162    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:12.698232 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.698184    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:12.698232 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.698209    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:12.698232 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.698218    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:12.698330 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.698190    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:12.698330 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.698265    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:12.700429 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.700412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.700515 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.700438    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:12.701125 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.701110    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:12.701191 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.701137    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:12.701191 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.701149    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:12.725501 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.725477    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-120.ec2.internal\" not found" node="ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.728878 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.728860    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-120.ec2.internal\" not found" node="ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.779452 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.779418    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:12.802528 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.802497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0389e2a0954ee4d1afb23ab5513a817-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal\" (UID: \"d0389e2a0954ee4d1afb23ab5513a817\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.802528 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.802526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0389e2a0954ee4d1afb23ab5513a817-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal\" (UID: \"d0389e2a0954ee4d1afb23ab5513a817\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.802685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.802543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e2c558c42ad344dfa74981a35dcbdcb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-120.ec2.internal\" (UID: \"1e2c558c42ad344dfa74981a35dcbdcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.879881 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.879848    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:12.903175 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.903150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0389e2a0954ee4d1afb23ab5513a817-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal\" (UID: \"d0389e2a0954ee4d1afb23ab5513a817\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.903263 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.903179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0389e2a0954ee4d1afb23ab5513a817-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal\" (UID: \"d0389e2a0954ee4d1afb23ab5513a817\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.903263 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.903209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e2c558c42ad344dfa74981a35dcbdcb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-120.ec2.internal\" (UID: \"1e2c558c42ad344dfa74981a35dcbdcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.903349 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.903281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0389e2a0954ee4d1afb23ab5513a817-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal\" (UID: \"d0389e2a0954ee4d1afb23ab5513a817\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.903349 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.903287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0389e2a0954ee4d1afb23ab5513a817-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal\" (UID: \"d0389e2a0954ee4d1afb23ab5513a817\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.903349 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:12.903287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e2c558c42ad344dfa74981a35dcbdcb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-120.ec2.internal\" (UID: \"1e2c558c42ad344dfa74981a35dcbdcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:12.980611 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:12.980536    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:13.027996 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:13.027970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:13.031512 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:13.031490    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:13.080925 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:13.080885    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:13.181405 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:13.181361    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:14.365325 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:14.365298    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:14.465803 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:14.465771    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:14.520324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.520292    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 19:39:14.520823 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.520502    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 19:39:14.566932 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:14.566766    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:14.626564 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.626484    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 19:34:13 +0000 UTC" deadline="2028-03-21 11:13:09.886781751 +0000 UTC"
Sep 16 19:39:14.626564 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.626521    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13239h33m55.26026602s"
Sep 16 19:39:14.667818 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:14.667784    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-120.ec2.internal\" not found"
Sep 16 19:39:14.697202 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.697176    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:14.698023 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.698006    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:14.698086 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.698039    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:14.698086 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.698053    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:14.708070 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.708051    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:14.734112 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:14.734077    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0389e2a0954ee4d1afb23ab5513a817.slice/crio-e99f8fe0c907d3c3db0ebca043d764aa76cd1ef8eaddcf8ae16bfc7dde0cd2c8 WatchSource:0}: Error finding container e99f8fe0c907d3c3db0ebca043d764aa76cd1ef8eaddcf8ae16bfc7dde0cd2c8: Status 404 returned error can't find the container with id e99f8fe0c907d3c3db0ebca043d764aa76cd1ef8eaddcf8ae16bfc7dde0cd2c8
Sep 16 19:39:14.734331 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:14.734311    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2c558c42ad344dfa74981a35dcbdcb.slice/crio-c32fda57fd2772c073572ae8590ca15bfb4df8685bfdcecc77c16bef1bb08d58 WatchSource:0}: Error finding container c32fda57fd2772c073572ae8590ca15bfb4df8685bfdcecc77c16bef1bb08d58: Status 404 returned error can't find the container with id c32fda57fd2772c073572ae8590ca15bfb4df8685bfdcecc77c16bef1bb08d58
Sep 16 19:39:14.737886 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.737873    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:39:14.784173 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.784150    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:14.784460 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.784445    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:14.798831 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.798813    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 19:39:14.799800 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.799786    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal"
Sep 16 19:39:14.809843 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:14.809829    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 19:39:15.065608 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.065533    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:15.562529 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.562453    2561 apiserver.go:52] "Watching apiserver"
Sep 16 19:39:15.570655 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.570622    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 19:39:15.571019 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.570995    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-zvqtq","openshift-ovn-kubernetes/ovnkube-node-lhw67","kube-system/global-pull-secret-syncer-wct9n","kube-system/konnectivity-agent-z26db","kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mkc5n","openshift-dns/node-resolver-fjkh7","openshift-image-registry/node-ca-5l96d","openshift-multus/multus-8znj5","openshift-multus/network-metrics-daemon-cc5h9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal","openshift-multus/multus-additional-cni-plugins-525g9","openshift-network-diagnostics/network-check-target-m5zz2"]
Sep 16 19:39:15.573688 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.573664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.575859 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.575831    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:15.575969 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.575925    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:15.576692 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.576670    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 19:39:15.576931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.576910    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 19:39:15.577140 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.577125    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wp6dc\""
Sep 16 19:39:15.578510 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.578491    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:15.578614 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.578543    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:15.580079 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.580061    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:15.582691 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.582637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.582852 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.582776    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 19:39:15.583118 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.583092    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 19:39:15.583200 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.583138    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-phw22\""
Sep 16 19:39:15.584950 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.584694    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 19:39:15.584950 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.584741    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.585465 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.585450    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wdgn8\""
Sep 16 19:39:15.585688 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.585667    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 19:39:15.585794 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.585781    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 19:39:15.585900 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.585866    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 19:39:15.587127 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.587106    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.587956 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.587937    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 19:39:15.588055 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.587969    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:39:15.588461 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.588439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q7qnw\""
Sep 16 19:39:15.589674 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.589604    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.590487 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.590074    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kdbbc\""
Sep 16 19:39:15.590487 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.590091    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 19:39:15.590487 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.590261    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 19:39:15.590487 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.590295    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 19:39:15.590757 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.590601    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 19:39:15.592669 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.592387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.592767 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.592689    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 19:39:15.593265 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.593246    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 19:39:15.593357 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.593268    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 19:39:15.593357 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.593269    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 19:39:15.593739 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.593720    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wlxtx\""
Sep 16 19:39:15.593816 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.593785    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 19:39:15.594720 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.594701    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 19:39:15.595486 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.595455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.595634 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.595605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-host\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.595705 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.595658    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-cnibin\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.595779 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.595765    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-socket-dir-parent\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.595832 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.595804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.595886 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.595864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-socket-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.595940 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.595892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnv6w\" (UniqueName: \"kubernetes.io/projected/8fa14a37-2fef-460b-bc21-b6875b34ba4d-kube-api-access-vnv6w\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.595940 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.595924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d721429-3ffb-43f2-ae20-6ee59561704b-tmp\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596043 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-cni-multus\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.596097 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-kubelet\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.596148 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf2tk\" (UniqueName: \"kubernetes.io/projected/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-kube-api-access-tf2tk\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.596203 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hq2\" (UniqueName: \"kubernetes.io/projected/ca3b10ac-d514-413b-aa81-23fbe19df3bd-kube-api-access-d5hq2\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:15.596203 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-kubernetes\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596300 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-sys\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596300 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4sn\" (UniqueName: \"kubernetes.io/projected/1d721429-3ffb-43f2-ae20-6ee59561704b-kube-api-access-wf4sn\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596300 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysconfig\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596507 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysctl-d\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596507 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-systemd\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596507 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-cni-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.596667 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-cni-binary-copy\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.596667 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-conf-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.596667 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-registration-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.596667 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.596667 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysctl-conf\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596897 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-run\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596897 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-lib-modules\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.596897 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-system-cni-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.596897 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-k8s-cni-cncf-io\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.596897 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:15.596897 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-device-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.596897 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-var-lib-kubelet\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-tuned\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bed5bc13-28aa-4988-a9ca-0bc86e03637e-hosts-file\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed5bc13-28aa-4988-a9ca-0bc86e03637e-tmp-dir\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425q7\" (UniqueName: \"kubernetes.io/projected/bed5bc13-28aa-4988-a9ca-0bc86e03637e-kube-api-access-425q7\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07142bb7-f90d-4bb3-b051-671f06731298-agent-certs\") pod \"konnectivity-agent-z26db\" (UID: \"07142bb7-f90d-4bb3-b051-671f06731298\") " pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597084    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-modprobe-d\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597132    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-cni-bin\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597157    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 19:39:15.597212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-hostroot\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.597640 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-daemon-config\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.597640 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:15.597640 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597350    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-sys-fs\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.597640 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-netns\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.597640 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-multus-certs\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.597640 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597437    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rdb8s\""
Sep 16 19:39:15.597640 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-etc-kubernetes\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.597640 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.597514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07142bb7-f90d-4bb3-b051-671f06731298-konnectivity-ca\") pod \"konnectivity-agent-z26db\" (UID: \"07142bb7-f90d-4bb3-b051-671f06731298\") " pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:15.598319 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.596856    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 19:39:15.598875 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.598853    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 19:39:15.598961 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.598877    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-65kkm\""
Sep 16 19:39:15.599130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.599114    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.599200 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.599162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 19:39:15.599405 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.599387    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:39:15.600937 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.600904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.601028 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.600977    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:15.602314 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.602162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 19:39:15.602314 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.602233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j8thk\""
Sep 16 19:39:15.603169 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.602906    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 19:39:15.603169 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.602983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 19:39:15.605819 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.605760    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 19:39:15.628144 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.628111    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9qxdc"
Sep 16 19:39:15.638398 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.638364    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9qxdc"
Sep 16 19:39:15.682216 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.682165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal" event={"ID":"1e2c558c42ad344dfa74981a35dcbdcb","Type":"ContainerStarted","Data":"c32fda57fd2772c073572ae8590ca15bfb4df8685bfdcecc77c16bef1bb08d58"}
Sep 16 19:39:15.684058 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.684034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal" event={"ID":"d0389e2a0954ee4d1afb23ab5513a817","Type":"ContainerStarted","Data":"e99f8fe0c907d3c3db0ebca043d764aa76cd1ef8eaddcf8ae16bfc7dde0cd2c8"}
Sep 16 19:39:15.685912 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.685891    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 19:39:15.698143 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736d0254-9446-4f6c-b40a-66c5a93c5c53-host-slash\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.698244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysconfig\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysctl-d\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-conf-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.698412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-kubelet\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.698412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-conf-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.698412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.698412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysconfig\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-cni-bin\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.698412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-cni-netd\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysctl-d\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysctl-conf\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-run\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-lib-modules\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-k8s-cni-cncf-io\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-sysctl-conf\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-run\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-k8s-cni-cncf-io\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02086a7b-8320-4264-96c3-fba73847dbce-kubelet-config\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-device-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-var-lib-kubelet\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.698738 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-lib-modules\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bed5bc13-28aa-4988-a9ca-0bc86e03637e-hosts-file\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425q7\" (UniqueName: \"kubernetes.io/projected/bed5bc13-28aa-4988-a9ca-0bc86e03637e-kube-api-access-425q7\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-device-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-systemd-units\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-node-log\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-cni-bin\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-hostroot\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-daemon-config\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.698993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-env-overrides\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cni-binary-copy\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-var-lib-kubelet\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-cni-bin\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-sys-fs\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-hostroot\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.699449 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-multus-certs\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-etc-kubernetes\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-etc-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.699252    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lstb\" (UniqueName: \"kubernetes.io/projected/bbcd8fa0-2baf-4439-961f-b629b3a1b899-kube-api-access-9lstb\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.699317    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:39:16.199297931 +0000 UTC m=+2.971168340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7494z\" (UniqueName: \"kubernetes.io/projected/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-kube-api-access-7494z\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7529a9e4-1769-414e-939e-79eae5b17d12-host\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-host\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwk5g\" (UniqueName: \"kubernetes.io/projected/736d0254-9446-4f6c-b40a-66c5a93c5c53-kube-api-access-fwk5g\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnv6w\" (UniqueName: \"kubernetes.io/projected/8fa14a37-2fef-460b-bc21-b6875b34ba4d-kube-api-access-vnv6w\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-kubelet\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-multus-certs\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf2tk\" (UniqueName: \"kubernetes.io/projected/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-kube-api-access-tf2tk\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-slash\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.700067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-systemd\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cnibin\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/736d0254-9446-4f6c-b40a-66c5a93c5c53-iptables-alerter-script\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-sys\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovnkube-script-lib\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-systemd\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-cni-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-cni-binary-copy\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-host\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovnkube-config\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovn-node-metrics-cert\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07142bb7-f90d-4bb3-b051-671f06731298-konnectivity-ca\") pod \"konnectivity-agent-z26db\" (UID: \"07142bb7-f90d-4bb3-b051-671f06731298\") " pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.699962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-registration-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-system-cni-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-tuned\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.700931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-kubelet\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed5bc13-28aa-4988-a9ca-0bc86e03637e-tmp-dir\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07142bb7-f90d-4bb3-b051-671f06731298-agent-certs\") pod \"konnectivity-agent-z26db\" (UID: \"07142bb7-f90d-4bb3-b051-671f06731298\") " pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-registration-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-systemd\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-modprobe-d\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-sys\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-etc-kubernetes\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bed5bc13-28aa-4988-a9ca-0bc86e03637e-hosts-file\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-sys-fs\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-daemon-config\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-cni-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-modprobe-d\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-system-cni-dir\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-system-cni-dir\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.701740 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-netns\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-run-netns\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02086a7b-8320-4264-96c3-fba73847dbce-dbus\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-cnibin\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-socket-dir-parent\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-var-lib-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bg88\" (UniqueName: \"kubernetes.io/projected/7529a9e4-1769-414e-939e-79eae5b17d12-kube-api-access-9bg88\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-socket-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d721429-3ffb-43f2-ae20-6ee59561704b-tmp\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.700993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-cni-binary-copy\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-run-netns\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-cni-multus\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07142bb7-f90d-4bb3-b051-671f06731298-konnectivity-ca\") pod \"konnectivity-agent-z26db\" (UID: \"07142bb7-f90d-4bb3-b051-671f06731298\") " pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-host-var-lib-cni-multus\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-cnibin\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-multus-socket-dir-parent\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.702551 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hq2\" (UniqueName: \"kubernetes.io/projected/ca3b10ac-d514-413b-aa81-23fbe19df3bd-kube-api-access-d5hq2\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fa14a37-2fef-460b-bc21-b6875b34ba4d-socket-dir\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7529a9e4-1769-414e-939e-79eae5b17d12-serviceca\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-kubernetes\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701325    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed5bc13-28aa-4988-a9ca-0bc86e03637e-tmp-dir\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701320    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-kubernetes\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4sn\" (UniqueName: \"kubernetes.io/projected/1d721429-3ffb-43f2-ae20-6ee59561704b-kube-api-access-wf4sn\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-ovn\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.703221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.701518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-log-socket\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.705808 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.705773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d721429-3ffb-43f2-ae20-6ee59561704b-tmp\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.705924 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.705835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d721429-3ffb-43f2-ae20-6ee59561704b-etc-tuned\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.706171 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.706150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07142bb7-f90d-4bb3-b051-671f06731298-agent-certs\") pod \"konnectivity-agent-z26db\" (UID: \"07142bb7-f90d-4bb3-b051-671f06731298\") " pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:15.709062 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.709021    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:15.709062 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.709046    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:15.709062 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.709059    2561 projected.go:196] Error preparing data for projected volume kube-api-access-q464f for pod openshift-network-diagnostics/network-check-target-m5zz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:15.709728 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.709111    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f podName:a742ff26-e9a8-4033-b661-c370dff3279f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:16.209096575 +0000 UTC m=+2.980966964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q464f" (UniqueName: "kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f") pod "network-check-target-m5zz2" (UID: "a742ff26-e9a8-4033-b661-c370dff3279f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:15.711196 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.711173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-425q7\" (UniqueName: \"kubernetes.io/projected/bed5bc13-28aa-4988-a9ca-0bc86e03637e-kube-api-access-425q7\") pod \"node-resolver-fjkh7\" (UID: \"bed5bc13-28aa-4988-a9ca-0bc86e03637e\") " pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.711593 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.711573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnv6w\" (UniqueName: \"kubernetes.io/projected/8fa14a37-2fef-460b-bc21-b6875b34ba4d-kube-api-access-vnv6w\") pod \"aws-ebs-csi-driver-node-2rfkv\" (UID: \"8fa14a37-2fef-460b-bc21-b6875b34ba4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.712841 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.712819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4sn\" (UniqueName: \"kubernetes.io/projected/1d721429-3ffb-43f2-ae20-6ee59561704b-kube-api-access-wf4sn\") pod \"tuned-mkc5n\" (UID: \"1d721429-3ffb-43f2-ae20-6ee59561704b\") " pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.713145 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.713124    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hq2\" (UniqueName: \"kubernetes.io/projected/ca3b10ac-d514-413b-aa81-23fbe19df3bd-kube-api-access-d5hq2\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:15.713838 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.713815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf2tk\" (UniqueName: \"kubernetes.io/projected/0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131-kube-api-access-tf2tk\") pod \"multus-8znj5\" (UID: \"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131\") " pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.802211 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-kubelet\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-cni-bin\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-cni-netd\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-kubelet\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-cni-netd\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02086a7b-8320-4264-96c3-fba73847dbce-kubelet-config\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-cni-bin\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-systemd-units\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-systemd-units\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02086a7b-8320-4264-96c3-fba73847dbce-kubelet-config\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.802523    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:15.802574    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret podName:02086a7b-8320-4264-96c3-fba73847dbce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:16.302558476 +0000 UTC m=+3.074428882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret") pod "global-pull-secret-syncer-wct9n" (UID: "02086a7b-8320-4264-96c3-fba73847dbce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-node-log\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-env-overrides\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cni-binary-copy\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-node-log\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-etc-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lstb\" (UniqueName: \"kubernetes.io/projected/bbcd8fa0-2baf-4439-961f-b629b3a1b899-kube-api-access-9lstb\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.802839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-etc-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7494z\" (UniqueName: \"kubernetes.io/projected/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-kube-api-access-7494z\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7529a9e4-1769-414e-939e-79eae5b17d12-host\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwk5g\" (UniqueName: \"kubernetes.io/projected/736d0254-9446-4f6c-b40a-66c5a93c5c53-kube-api-access-fwk5g\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-slash\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-systemd\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.802990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cnibin\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/736d0254-9446-4f6c-b40a-66c5a93c5c53-iptables-alerter-script\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-slash\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovnkube-script-lib\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7529a9e4-1769-414e-939e-79eae5b17d12-host\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovnkube-config\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovn-node-metrics-cert\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-env-overrides\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cni-binary-copy\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.803573 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-systemd\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-system-cni-dir\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cnibin\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-run-netns\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02086a7b-8320-4264-96c3-fba73847dbce-dbus\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803544    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-system-cni-dir\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-var-lib-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803613    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02086a7b-8320-4264-96c3-fba73847dbce-dbus\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bg88\" (UniqueName: \"kubernetes.io/projected/7529a9e4-1769-414e-939e-79eae5b17d12-kube-api-access-9bg88\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-host-run-netns\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/736d0254-9446-4f6c-b40a-66c5a93c5c53-iptables-alerter-script\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7529a9e4-1769-414e-939e-79eae5b17d12-serviceca\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-var-lib-openvswitch\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovnkube-config\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803732    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-ovn\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.804324 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-log-socket\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.805130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736d0254-9446-4f6c-b40a-66c5a93c5c53-host-slash\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.805130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovnkube-script-lib\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.805130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-run-ovn\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.805130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736d0254-9446-4f6c-b40a-66c5a93c5c53-host-slash\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.805130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.803868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbcd8fa0-2baf-4439-961f-b629b3a1b899-log-socket\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.805130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.804098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7529a9e4-1769-414e-939e-79eae5b17d12-serviceca\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.805130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.804141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.805130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.804172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.806124 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.806098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbcd8fa0-2baf-4439-961f-b629b3a1b899-ovn-node-metrics-cert\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.825473 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.825409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bg88\" (UniqueName: \"kubernetes.io/projected/7529a9e4-1769-414e-939e-79eae5b17d12-kube-api-access-9bg88\") pod \"node-ca-5l96d\" (UID: \"7529a9e4-1769-414e-939e-79eae5b17d12\") " pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.825630 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.825609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7494z\" (UniqueName: \"kubernetes.io/projected/15ecb2f2-b01c-4588-b82a-a8f6bde8d89c-kube-api-access-7494z\") pod \"multus-additional-cni-plugins-525g9\" (UID: \"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c\") " pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.827089 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.827060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwk5g\" (UniqueName: \"kubernetes.io/projected/736d0254-9446-4f6c-b40a-66c5a93c5c53-kube-api-access-fwk5g\") pod \"iptables-alerter-zvqtq\" (UID: \"736d0254-9446-4f6c-b40a-66c5a93c5c53\") " pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.831548 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.831517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lstb\" (UniqueName: \"kubernetes.io/projected/bbcd8fa0-2baf-4439-961f-b629b3a1b899-kube-api-access-9lstb\") pod \"ovnkube-node-lhw67\" (UID: \"bbcd8fa0-2baf-4439-961f-b629b3a1b899\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.886627 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.886592    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fjkh7"
Sep 16 19:39:15.895880 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.895857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:15.908761 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.908731    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv"
Sep 16 19:39:15.914549 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.914535    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mkc5n"
Sep 16 19:39:15.923033 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.923011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8znj5"
Sep 16 19:39:15.930851 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.930829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:15.938457 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.938440    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-525g9"
Sep 16 19:39:15.947062 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.947044    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zvqtq"
Sep 16 19:39:15.953660 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.953640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5l96d"
Sep 16 19:39:15.973890 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:15.973871    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:16.207399 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.207320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:16.207533 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.207455    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:16.207533 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.207515    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:39:17.207501942 +0000 UTC m=+3.979372330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:16.308187 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.308162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:16.308384 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.308195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:16.308384 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.308320    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:16.308502 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.308387    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret podName:02086a7b-8320-4264-96c3-fba73847dbce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:17.308359114 +0000 UTC m=+4.080229503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret") pod "global-pull-secret-syncer-wct9n" (UID: "02086a7b-8320-4264-96c3-fba73847dbce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:16.308502 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.308322    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:16.308502 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.308412    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:16.308502 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.308425    2561 projected.go:196] Error preparing data for projected volume kube-api-access-q464f for pod openshift-network-diagnostics/network-check-target-m5zz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:16.308502 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.308466    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f podName:a742ff26-e9a8-4033-b661-c370dff3279f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:17.308455377 +0000 UTC m=+4.080325779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q464f" (UniqueName: "kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f") pod "network-check-target-m5zz2" (UID: "a742ff26-e9a8-4033-b661-c370dff3279f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:16.638753 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:16.638726    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07142bb7_f90d_4bb3_b051_671f06731298.slice/crio-12f6cf24dbefac2f75ed639f05664ac8a3fad53becceab8929f79cd91ab52d15 WatchSource:0}: Error finding container 12f6cf24dbefac2f75ed639f05664ac8a3fad53becceab8929f79cd91ab52d15: Status 404 returned error can't find the container with id 12f6cf24dbefac2f75ed639f05664ac8a3fad53becceab8929f79cd91ab52d15
Sep 16 19:39:16.639860 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:16.639776    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed5bc13_28aa_4988_a9ca_0bc86e03637e.slice/crio-db6f32b9394c591ebe9c4d689916b5ae62edcfc3f40c7d988090e1b4d7b7da54 WatchSource:0}: Error finding container db6f32b9394c591ebe9c4d689916b5ae62edcfc3f40c7d988090e1b4d7b7da54: Status 404 returned error can't find the container with id db6f32b9394c591ebe9c4d689916b5ae62edcfc3f40c7d988090e1b4d7b7da54
Sep 16 19:39:16.641509 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.641300    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:34:15 +0000 UTC" deadline="2028-06-06 04:32:56.065055048 +0000 UTC"
Sep 16 19:39:16.641509 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.641330    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15080h53m39.423728627s"
Sep 16 19:39:16.643000 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:16.642974    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d721429_3ffb_43f2_ae20_6ee59561704b.slice/crio-369cf8b55333bf9d9ac337baa719bd692aab049997d51ebb45829fb7c9294db7 WatchSource:0}: Error finding container 369cf8b55333bf9d9ac337baa719bd692aab049997d51ebb45829fb7c9294db7: Status 404 returned error can't find the container with id 369cf8b55333bf9d9ac337baa719bd692aab049997d51ebb45829fb7c9294db7
Sep 16 19:39:16.644250 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:16.644230    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8ea2e8_1fbc_4e09_97b7_f2b56caa7131.slice/crio-055f5132fb0ecb04078924b00140007aa6a054e15d5f414c7ff94f23d076282a WatchSource:0}: Error finding container 055f5132fb0ecb04078924b00140007aa6a054e15d5f414c7ff94f23d076282a: Status 404 returned error can't find the container with id 055f5132fb0ecb04078924b00140007aa6a054e15d5f414c7ff94f23d076282a
Sep 16 19:39:16.645587 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:16.645564    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ecb2f2_b01c_4588_b82a_a8f6bde8d89c.slice/crio-137b6c6f2b9eedd37ed388d10bc6b4350975691d2a73a97bbde9d343ef022c14 WatchSource:0}: Error finding container 137b6c6f2b9eedd37ed388d10bc6b4350975691d2a73a97bbde9d343ef022c14: Status 404 returned error can't find the container with id 137b6c6f2b9eedd37ed388d10bc6b4350975691d2a73a97bbde9d343ef022c14
Sep 16 19:39:16.646319 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:16.646299    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcd8fa0_2baf_4439_961f_b629b3a1b899.slice/crio-2a22902d0f9cb3f9293d1b33b94f927bdf66fefa2300939657744c15c5fa6aa5 WatchSource:0}: Error finding container 2a22902d0f9cb3f9293d1b33b94f927bdf66fefa2300939657744c15c5fa6aa5: Status 404 returned error can't find the container with id 2a22902d0f9cb3f9293d1b33b94f927bdf66fefa2300939657744c15c5fa6aa5
Sep 16 19:39:16.678056 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.678029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:16.678176 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:16.678152    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:16.689366 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.689336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"2a22902d0f9cb3f9293d1b33b94f927bdf66fefa2300939657744c15c5fa6aa5"}
Sep 16 19:39:16.691693 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.691655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerStarted","Data":"137b6c6f2b9eedd37ed388d10bc6b4350975691d2a73a97bbde9d343ef022c14"}
Sep 16 19:39:16.692878 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.692852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mkc5n" event={"ID":"1d721429-3ffb-43f2-ae20-6ee59561704b","Type":"ContainerStarted","Data":"369cf8b55333bf9d9ac337baa719bd692aab049997d51ebb45829fb7c9294db7"}
Sep 16 19:39:16.693931 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.693905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fjkh7" event={"ID":"bed5bc13-28aa-4988-a9ca-0bc86e03637e","Type":"ContainerStarted","Data":"db6f32b9394c591ebe9c4d689916b5ae62edcfc3f40c7d988090e1b4d7b7da54"}
Sep 16 19:39:16.695044 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.695017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv" event={"ID":"8fa14a37-2fef-460b-bc21-b6875b34ba4d","Type":"ContainerStarted","Data":"b1cb0f6a8fff78b8f2331c556cf426fba62445aa686c5a6c8f6fdff8fe599986"}
Sep 16 19:39:16.696231 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.696211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zvqtq" event={"ID":"736d0254-9446-4f6c-b40a-66c5a93c5c53","Type":"ContainerStarted","Data":"2727aeb3dfeecdfb85edee367b098463503f410ce9160f7d50c5abc645748de6"}
Sep 16 19:39:16.697394 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.697358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5l96d" event={"ID":"7529a9e4-1769-414e-939e-79eae5b17d12","Type":"ContainerStarted","Data":"f5b4a2331ad998fb4e2d6b9e22a6859e84f915c8dc70dffd006d5f69c2d0e173"}
Sep 16 19:39:16.698895 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.698866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8znj5" event={"ID":"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131","Type":"ContainerStarted","Data":"055f5132fb0ecb04078924b00140007aa6a054e15d5f414c7ff94f23d076282a"}
Sep 16 19:39:16.704626 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:16.704601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z26db" event={"ID":"07142bb7-f90d-4bb3-b051-671f06731298","Type":"ContainerStarted","Data":"12f6cf24dbefac2f75ed639f05664ac8a3fad53becceab8929f79cd91ab52d15"}
Sep 16 19:39:17.214879 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.214212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:17.214879 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.214403    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:17.214879 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.214472    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:39:19.21445314 +0000 UTC m=+5.986323532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:17.315148 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.315106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:17.315335 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.315161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:17.315335 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.315301    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:17.315479 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.315336    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:17.315479 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.315368    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:17.315479 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.315399    2561 projected.go:196] Error preparing data for projected volume kube-api-access-q464f for pod openshift-network-diagnostics/network-check-target-m5zz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:17.315479 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.315383    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret podName:02086a7b-8320-4264-96c3-fba73847dbce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:19.31535337 +0000 UTC m=+6.087223763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret") pod "global-pull-secret-syncer-wct9n" (UID: "02086a7b-8320-4264-96c3-fba73847dbce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:17.315479 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.315471    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f podName:a742ff26-e9a8-4033-b661-c370dff3279f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:19.315450711 +0000 UTC m=+6.087321119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q464f" (UniqueName: "kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f") pod "network-check-target-m5zz2" (UID: "a742ff26-e9a8-4033-b661-c370dff3279f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:17.643284 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.643120    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:34:15 +0000 UTC" deadline="2028-05-08 05:25:14.541137906 +0000 UTC"
Sep 16 19:39:17.643284 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.643179    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14385h45m56.897962808s"
Sep 16 19:39:17.677585 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.677520    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:17.677764 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.677648    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:17.680138 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.680113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:17.680260 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:17.680211    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:17.718463 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.718427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal" event={"ID":"1e2c558c42ad344dfa74981a35dcbdcb","Type":"ContainerStarted","Data":"b8a36acb8f25c65d907d86298bf347f50311a8b1824eb8e7d9d988918b71bba2"}
Sep 16 19:39:17.723592 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.723562    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0389e2a0954ee4d1afb23ab5513a817" containerID="abdac315c490a81431176272b1bf5e6524a9f3a53958907ed94d1d4d24aca6f8" exitCode=0
Sep 16 19:39:17.723741 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.723603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal" event={"ID":"d0389e2a0954ee4d1afb23ab5513a817","Type":"ContainerDied","Data":"abdac315c490a81431176272b1bf5e6524a9f3a53958907ed94d1d4d24aca6f8"}
Sep 16 19:39:17.759069 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:17.759017    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-120.ec2.internal" podStartSLOduration=3.7589974379999997 podStartE2EDuration="3.758997438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:17.738528026 +0000 UTC m=+4.510398441" watchObservedRunningTime="2026-09-16 19:39:17.758997438 +0000 UTC m=+4.530867850"
Sep 16 19:39:18.678271 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:18.678025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:18.678734 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:18.678402    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:18.754675 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:18.754635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal" event={"ID":"d0389e2a0954ee4d1afb23ab5513a817","Type":"ContainerStarted","Data":"2cd4ce26cc3b9d5aa704571014b80c544183e1c51683b06a463610e67db21795"}
Sep 16 19:39:19.229745 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:19.229707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:19.229934 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.229847    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:19.229934 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.229923    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:39:23.229904678 +0000 UTC m=+10.001775072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:19.330750 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:19.330709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:19.330916 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:19.330766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:19.330968 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.330938    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:19.331019 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.331002    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret podName:02086a7b-8320-4264-96c3-fba73847dbce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:23.330984266 +0000 UTC m=+10.102854676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret") pod "global-pull-secret-syncer-wct9n" (UID: "02086a7b-8320-4264-96c3-fba73847dbce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:19.331103 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.331094    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:19.331158 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.331113    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:19.331158 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.331125    2561 projected.go:196] Error preparing data for projected volume kube-api-access-q464f for pod openshift-network-diagnostics/network-check-target-m5zz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:19.331261 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.331161    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f podName:a742ff26-e9a8-4033-b661-c370dff3279f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:23.331149596 +0000 UTC m=+10.103020001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q464f" (UniqueName: "kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f") pod "network-check-target-m5zz2" (UID: "a742ff26-e9a8-4033-b661-c370dff3279f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:19.680428 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:19.680341    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:19.680862 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.680486    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:19.680862 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:19.680856    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:19.680965 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:19.680951    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:20.677907 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:20.677873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:20.678076 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:20.677988    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:21.678051 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:21.678016    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:21.678472 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:21.678030    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:21.678472 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:21.678169    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:21.678472 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:21.678221    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:22.677802 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:22.677697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:22.678006 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:22.677856    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:23.267228 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:23.267098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:23.267728 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.267264    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:23.267728 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.267331    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:39:31.267311852 +0000 UTC m=+18.039182254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:23.368186 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:23.368146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:23.368352 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:23.368196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:23.368450 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.368355    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:23.368450 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.368439    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret podName:02086a7b-8320-4264-96c3-fba73847dbce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:31.368421203 +0000 UTC m=+18.140291612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret") pod "global-pull-secret-syncer-wct9n" (UID: "02086a7b-8320-4264-96c3-fba73847dbce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:23.368737 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.368711    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:23.368737 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.368741    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:23.368737 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.368754    2561 projected.go:196] Error preparing data for projected volume kube-api-access-q464f for pod openshift-network-diagnostics/network-check-target-m5zz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:23.368976 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.368805    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f podName:a742ff26-e9a8-4033-b661-c370dff3279f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:31.36879083 +0000 UTC m=+18.140661236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q464f" (UniqueName: "kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f") pod "network-check-target-m5zz2" (UID: "a742ff26-e9a8-4033-b661-c370dff3279f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:23.677866 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:23.677790    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:23.678018 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.677898    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:23.678018 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:23.677976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:23.678129 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:23.678068    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:24.677539 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:24.677502    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:24.677978 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:24.677655    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:25.678042 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:25.678011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:25.678489 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:25.678134    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:25.678489 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:25.678202    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:25.678489 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:25.678322    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:26.677700 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:26.677665    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:26.677885 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:26.677785    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:27.677823 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:27.677785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:27.678280 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:27.677833    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:27.678280 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:27.677930    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:27.678280 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:27.678068    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:28.677058 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:28.677021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:28.677213 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:28.677137    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:29.678152 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:29.678120    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:29.678645 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:29.678256    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:29.678645 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:29.678295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:29.678645 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:29.678417    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:30.677651 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:30.677619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:30.677846 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:30.677726    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:31.324387 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:31.324331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:31.324784 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.324461    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:31.324784 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.324537    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:39:47.324520874 +0000 UTC m=+34.096391263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:31.425189 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:31.425148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:31.425189 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:31.425198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:31.425421 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.425335    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:31.425421 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.425336    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:31.425421 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.425413    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:31.425421 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.425418    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret podName:02086a7b-8320-4264-96c3-fba73847dbce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:47.425399971 +0000 UTC m=+34.197270360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret") pod "global-pull-secret-syncer-wct9n" (UID: "02086a7b-8320-4264-96c3-fba73847dbce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:31.425421 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.425423    2561 projected.go:196] Error preparing data for projected volume kube-api-access-q464f for pod openshift-network-diagnostics/network-check-target-m5zz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:31.425640 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.425476    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f podName:a742ff26-e9a8-4033-b661-c370dff3279f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:47.425451 +0000 UTC m=+34.197321402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q464f" (UniqueName: "kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f") pod "network-check-target-m5zz2" (UID: "a742ff26-e9a8-4033-b661-c370dff3279f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:31.677975 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:31.677893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:31.678134 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.678010    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:31.678134 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:31.678060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:31.678240 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:31.678168    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:32.677650 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:32.677611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:32.678072 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:32.677773    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:33.677844 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:33.677807    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:33.678280 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:33.677924    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:33.678280 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:33.677980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:33.678280 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:33.678083    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:34.677102 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.677082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:34.677466 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:34.677224    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:34.792978 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.789169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fjkh7" event={"ID":"bed5bc13-28aa-4988-a9ca-0bc86e03637e","Type":"ContainerStarted","Data":"e87d4f15a011c838afcdfa9ad8a210a9a639a160bd4dde47cc8dc7405bc94149"}
Sep 16 19:39:34.795918 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.795896    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv" event={"ID":"8fa14a37-2fef-460b-bc21-b6875b34ba4d","Type":"ContainerStarted","Data":"ba444afbbbf4ef40167a70fa5bf72f9a454537e2e28b3ce7192ed3e71b7aeb26"}
Sep 16 19:39:34.797500 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.797448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5l96d" event={"ID":"7529a9e4-1769-414e-939e-79eae5b17d12","Type":"ContainerStarted","Data":"aa8b2e261e86907febfc33cc7fa00f422ee91a857e0e459e0bc42ad8eff5a242"}
Sep 16 19:39:34.799315 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.799289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z26db" event={"ID":"07142bb7-f90d-4bb3-b051-671f06731298","Type":"ContainerStarted","Data":"e0089b11711913828023433bac50d2d52a3a9a67ad362a23483edb90afc80942"}
Sep 16 19:39:34.804287 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.802760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"3fb0f4b860d750e3278d8e4c9989e6f936bd13dd843fddaca357c2101e6d9fa8"}
Sep 16 19:39:34.804287 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.802832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"821c2c72d841c2d6e526c101590e5ed99387692a91133a8f481b10bdc4bb9303"}
Sep 16 19:39:34.804287 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.803540    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fjkh7" podStartSLOduration=3.998311418 podStartE2EDuration="21.803525435s" totalImagesPullingTime="17.805214017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:34.803082035 +0000 UTC m=+21.574952447" watchObservedRunningTime="2026-09-16 19:39:34.803525435 +0000 UTC m=+21.575395849"
Sep 16 19:39:34.804287 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.803802    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-120.ec2.internal" podStartSLOduration=20.803793992 podStartE2EDuration="20.803793992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:18.771635411 +0000 UTC m=+5.543505822" watchObservedRunningTime="2026-09-16 19:39:34.803793992 +0000 UTC m=+21.575664401"
Sep 16 19:39:34.806839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.806804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerStarted","Data":"206f26952547611a411a91a324e44af43326a130957424bdec370f8d7d5236a1"}
Sep 16 19:39:34.808777 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.808754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mkc5n" event={"ID":"1d721429-3ffb-43f2-ae20-6ee59561704b","Type":"ContainerStarted","Data":"a8d17d18962fba8a0aafc3ca6bfbc628eb2117ef9e2bbd939d82029e7d7bbde4"}
Sep 16 19:39:34.837708 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.837655    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z26db" podStartSLOduration=11.480874831 podStartE2EDuration="21.837638989s" totalImagesPullingTime="10.356764158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:34.822668258 +0000 UTC m=+21.594538669" watchObservedRunningTime="2026-09-16 19:39:34.837638989 +0000 UTC m=+21.609509400"
Sep 16 19:39:34.837873 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.837843    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5l96d" podStartSLOduration=3.040479803 podStartE2EDuration="20.837836285s" totalImagesPullingTime="17.797356482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:34.837090134 +0000 UTC m=+21.608960545" watchObservedRunningTime="2026-09-16 19:39:34.837836285 +0000 UTC m=+21.609706722"
Sep 16 19:39:34.876214 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:34.876151    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mkc5n" podStartSLOduration=4.072517739 podStartE2EDuration="21.876130911s" totalImagesPullingTime="17.803613172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:34.875770519 +0000 UTC m=+21.647640930" watchObservedRunningTime="2026-09-16 19:39:34.876130911 +0000 UTC m=+21.648001323"
Sep 16 19:39:35.680684 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.680504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:35.680825 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.680504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:35.680825 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:35.680753    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:35.680933 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:35.680852    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:35.799362 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.799342    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 19:39:35.812810 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.812785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv" event={"ID":"8fa14a37-2fef-460b-bc21-b6875b34ba4d","Type":"ContainerStarted","Data":"14023f93f21dc3b4877f79556451081c84301055cf8848c09b9b08bcd114924a"}
Sep 16 19:39:35.814198 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.814176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zvqtq" event={"ID":"736d0254-9446-4f6c-b40a-66c5a93c5c53","Type":"ContainerStarted","Data":"c1bb09639f43a98d37efbc5ae27108fa159d85a375ac39fb6cfa429c63a1c02b"}
Sep 16 19:39:35.815715 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.815696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8znj5" event={"ID":"0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131","Type":"ContainerStarted","Data":"cce4485787578120160ea38035eb179e557a23d14c9ac96c8024931f6e62e7bc"}
Sep 16 19:39:35.818320 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.818304    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:39:35.818644 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.818623    2561 generic.go:356] "Generic (PLEG): container finished" podID="bbcd8fa0-2baf-4439-961f-b629b3a1b899" containerID="3fb0f4b860d750e3278d8e4c9989e6f936bd13dd843fddaca357c2101e6d9fa8" exitCode=1
Sep 16 19:39:35.818734 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.818692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerDied","Data":"3fb0f4b860d750e3278d8e4c9989e6f936bd13dd843fddaca357c2101e6d9fa8"}
Sep 16 19:39:35.818788 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.818773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"1b3618e924c1320aa8e9392e12216f010b606d941ae654c12a0f6277e4b39915"}
Sep 16 19:39:35.818839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.818825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"5bbb665b559e91a54bdb5cc2dfa319b9b9623d9007aae2b9fd6d6e589041a650"}
Sep 16 19:39:35.818892 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.818880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"5dec085f672a1d619422ede59890d88e2693cb17610d1b6eb6b235067ba5b0ae"}
Sep 16 19:39:35.818928 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.818922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"c274ffab59cbf842e7b8d4e23f2b354c7693eef8e3730ee82871a206eeb7095c"}
Sep 16 19:39:35.820123 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.820105    2561 generic.go:356] "Generic (PLEG): container finished" podID="15ecb2f2-b01c-4588-b82a-a8f6bde8d89c" containerID="206f26952547611a411a91a324e44af43326a130957424bdec370f8d7d5236a1" exitCode=0
Sep 16 19:39:35.820218 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.820168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerDied","Data":"206f26952547611a411a91a324e44af43326a130957424bdec370f8d7d5236a1"}
Sep 16 19:39:35.832689 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.831409    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zvqtq" podStartSLOduration=4.035328878 podStartE2EDuration="21.831392898s" totalImagesPullingTime="17.79606402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:35.830419635 +0000 UTC m=+22.602290049" watchObservedRunningTime="2026-09-16 19:39:35.831392898 +0000 UTC m=+22.603263303"
Sep 16 19:39:35.851496 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:35.851449    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8znj5" podStartSLOduration=4.743652683 podStartE2EDuration="22.851436201s" totalImagesPullingTime="18.107783518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:35.85135029 +0000 UTC m=+22.623220701" watchObservedRunningTime="2026-09-16 19:39:35.851436201 +0000 UTC m=+22.623306612"
Sep 16 19:39:36.154709 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:36.154676    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:36.155276 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:36.155256    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:36.643302 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:36.643180    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T19:39:35.799358207Z","UUID":"34a46ae9-cf40-4070-8012-22b7ed555429","Handler":null,"Name":"","Endpoint":""}
Sep 16 19:39:36.645210 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:36.645182    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 19:39:36.645345 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:36.645219    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 19:39:36.677554 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:36.677527    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:36.677682 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:36.677663    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:36.825450 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:36.825413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv" event={"ID":"8fa14a37-2fef-460b-bc21-b6875b34ba4d","Type":"ContainerStarted","Data":"ba1fd0d1e6cb01146131794aeae7dea7d17742292f3661b623143421962b99e6"}
Sep 16 19:39:36.861664 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:36.861571    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2rfkv" podStartSLOduration=4.281983488 podStartE2EDuration="23.861554404s" totalImagesPullingTime="19.579570916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:36.860937693 +0000 UTC m=+23.632808107" watchObservedRunningTime="2026-09-16 19:39:36.861554404 +0000 UTC m=+23.633424818"
Sep 16 19:39:37.677624 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:37.677588    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:37.677913 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:37.677722    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:37.677913 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:37.677781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:37.677913 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:37.677873    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:37.831754 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:37.831729    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:39:37.832165 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:37.832113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"b42125996a648d068ffb57d498693de56587e6ef9a9b4b648c14f216087125c7"}
Sep 16 19:39:37.832231 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:37.832210    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 19:39:38.678055 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:38.678019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:38.678239 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:38.678139    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:39.677638 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:39.677558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:39.677989 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:39.677575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:39.677989 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:39.677697    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:39.677989 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:39.677731    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:40.677886 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.677849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:40.678503 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:40.677962    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:40.842701 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.842674    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:39:40.843058 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.843033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"bcb802ae8564b502a29747e5c94c4e5e47a4046ba73b226b26521bb18c77a98d"}
Sep 16 19:39:40.843328 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.843304    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:40.843439 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.843398    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:40.843560 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.843543    2561 scope.go:117] "RemoveContainer" containerID="3fb0f4b860d750e3278d8e4c9989e6f936bd13dd843fddaca357c2101e6d9fa8"
Sep 16 19:39:40.845069 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.845046    2561 generic.go:356] "Generic (PLEG): container finished" podID="15ecb2f2-b01c-4588-b82a-a8f6bde8d89c" containerID="a94d5ae70f810e4147aab58f3263387e4f2d971a7653331015d5cee7d211b72b" exitCode=0
Sep 16 19:39:40.845182 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.845078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerDied","Data":"a94d5ae70f810e4147aab58f3263387e4f2d971a7653331015d5cee7d211b72b"}
Sep 16 19:39:40.859235 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:40.859212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:41.677358 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.677323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:41.677501 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:41.677449    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:41.677544 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.677500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:41.677597 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:41.677580    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:41.773281 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.773248    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wct9n"]
Sep 16 19:39:41.776118 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.776090    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cc5h9"]
Sep 16 19:39:41.776252 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.776201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:41.776310 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:41.776287    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:41.776724 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.776700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m5zz2"]
Sep 16 19:39:41.850369 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.850340    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:39:41.850746 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.850725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" event={"ID":"bbcd8fa0-2baf-4439-961f-b629b3a1b899","Type":"ContainerStarted","Data":"c081fe9c75f00f000b1b43cc08cae2ac0358d2a4d1583044e2c9f5a65f2614ab"}
Sep 16 19:39:41.850918 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.850899    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:41.853004 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.852985    2561 generic.go:356] "Generic (PLEG): container finished" podID="15ecb2f2-b01c-4588-b82a-a8f6bde8d89c" containerID="6e0db955270cadad8efef29f215e140dd1653f55fc2597928266d2757540e30f" exitCode=0
Sep 16 19:39:41.853094 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.853048    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:41.853094 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.853046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerDied","Data":"6e0db955270cadad8efef29f215e140dd1653f55fc2597928266d2757540e30f"}
Sep 16 19:39:41.853177 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.853094    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:41.853212 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:41.853190    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:41.853254 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:41.853220    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:41.865893 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.865876    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:39:41.883720 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:41.883681    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67" podStartSLOduration=10.049047597 podStartE2EDuration="27.88366706s" totalImagesPullingTime="17.834619463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:41.883438122 +0000 UTC m=+28.655308534" watchObservedRunningTime="2026-09-16 19:39:41.88366706 +0000 UTC m=+28.655537690"
Sep 16 19:39:42.858240 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:42.857981    2561 generic.go:356] "Generic (PLEG): container finished" podID="15ecb2f2-b01c-4588-b82a-a8f6bde8d89c" containerID="4136dad1b1f3de39f575f835bb59bc5925904b672fe843350c073e1bc815c9c9" exitCode=0
Sep 16 19:39:42.858240 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:42.858060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerDied","Data":"4136dad1b1f3de39f575f835bb59bc5925904b672fe843350c073e1bc815c9c9"}
Sep 16 19:39:43.272032 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:43.272001    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:43.272192 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:43.272114    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 19:39:43.272621 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:43.272601    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z26db"
Sep 16 19:39:43.678237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:43.678171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:43.678412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:43.678291    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:43.678412 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:43.678321    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:43.678412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:43.678335    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:43.678557 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:43.678420    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:43.678557 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:43.678476    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:45.677365 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:45.677322    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:45.677890 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:45.677322    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:45.677890 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:45.677322    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:45.677890 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:45.677469    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wct9n" podUID="02086a7b-8320-4264-96c3-fba73847dbce"
Sep 16 19:39:45.677890 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:45.677565    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5zz2" podUID="a742ff26-e9a8-4033-b661-c370dff3279f"
Sep 16 19:39:45.677890 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:45.677664    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:39:47.345256 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.345191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:47.345631 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.345295    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:47.345631 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.345343    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:40:19.345331501 +0000 UTC m=+66.117201890 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:47.446487 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.446448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:47.446487 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.446485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:47.446693 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.446598    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:47.446693 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.446610    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:47.446693 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.446629    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:47.446693 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.446640    2561 projected.go:196] Error preparing data for projected volume kube-api-access-q464f for pod openshift-network-diagnostics/network-check-target-m5zz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:47.446693 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.446650    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret podName:02086a7b-8320-4264-96c3-fba73847dbce nodeName:}" failed. No retries permitted until 2026-09-16 19:40:19.446635093 +0000 UTC m=+66.218505502 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret") pod "global-pull-secret-syncer-wct9n" (UID: "02086a7b-8320-4264-96c3-fba73847dbce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:47.446693 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.446677    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f podName:a742ff26-e9a8-4033-b661-c370dff3279f nodeName:}" failed. No retries permitted until 2026-09-16 19:40:19.446666015 +0000 UTC m=+66.218536404 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q464f" (UniqueName: "kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f") pod "network-check-target-m5zz2" (UID: "a742ff26-e9a8-4033-b661-c370dff3279f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:47.526061 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.526038    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-120.ec2.internal" event="NodeReady"
Sep 16 19:39:47.526229 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.526153    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 19:39:47.577289 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.577259    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-869d7dbbbf-8k7x9"]
Sep 16 19:39:47.593213 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.593186    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"]
Sep 16 19:39:47.593354 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.593302    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.596442 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.596365    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 19:39:47.596442 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.596423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 19:39:47.596600 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.596446    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 19:39:47.596692 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.596680    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dtnsg\""
Sep 16 19:39:47.602189 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.601996    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 19:39:47.605471 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.605450    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"]
Sep 16 19:39:47.605584 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.605569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.609987 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.609965    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 19:39:47.610176 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.610157    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 19:39:47.610280 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.610155    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 19:39:47.610280 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.610233    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 19:39:47.610280 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.610262    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 19:39:47.610280 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.610276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 19:39:47.610601 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.610584    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 19:39:47.621703 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.621680    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"]
Sep 16 19:39:47.621807 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.621789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:39:47.624523 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.624488    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xnfsp\""
Sep 16 19:39:47.624523 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.624500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 19:39:47.634054 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.634027    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869d7dbbbf-8k7x9"]
Sep 16 19:39:47.634143 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.634064    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"]
Sep 16 19:39:47.634143 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.634077    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"]
Sep 16 19:39:47.634143 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.634099    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"]
Sep 16 19:39:47.634143 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.634113    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dn4vj"]
Sep 16 19:39:47.634330 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.634151    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.639690 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.639671    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 19:39:47.648666 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.648648    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ndpfg"]
Sep 16 19:39:47.648781 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.648765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn4vj"
Sep 16 19:39:47.654200 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.654182    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 19:39:47.654408 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.654394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 19:39:47.654466 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.654451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tf6pp\""
Sep 16 19:39:47.654645 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.654632    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 19:39:47.660247 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.660229    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dn4vj"]
Sep 16 19:39:47.660342 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.660255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ndpfg"]
Sep 16 19:39:47.660416 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.660354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.663720 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.663703    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 19:39:47.663834 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.663730    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 19:39:47.663920 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.663906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdb5m\""
Sep 16 19:39:47.677904 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.677876    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:39:47.678004 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.677883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:39:47.678139 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.678124    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:39:47.680498 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.680482    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmvpx\""
Sep 16 19:39:47.680760 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.680735    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 19:39:47.680843 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.680741    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 19:39:47.680843 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.680740    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 19:39:47.681292 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.681277    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 19:39:47.681409 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.681344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2djhj\""
Sep 16 19:39:47.749544 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed7aea78-8e03-41ef-b027-ef55194572c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d56d5995d-8447t\" (UID: \"ed7aea78-8e03-41ef-b027-ef55194572c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:39:47.749544 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa-cert\") pod \"ingress-canary-dn4vj\" (UID: \"cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa\") " pod="openshift-ingress-canary/ingress-canary-dn4vj"
Sep 16 19:39:47.749737 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.749737 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-hub\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.749737 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-tmp\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.749737 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mt6k\" (UniqueName: \"kubernetes.io/projected/c42e50fb-1452-4a79-bf3d-ccaf42264113-kube-api-access-7mt6k\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.749737 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-bound-sa-token\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.749737 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zwf\" (UniqueName: \"kubernetes.io/projected/ed7aea78-8e03-41ef-b027-ef55194572c1-kube-api-access-z6zwf\") pod \"managed-serviceaccount-addon-agent-d56d5995d-8447t\" (UID: \"ed7aea78-8e03-41ef-b027-ef55194572c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:39:47.749737 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-klusterlet-config\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.749946 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqxg\" (UniqueName: \"kubernetes.io/projected/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-kube-api-access-jbqxg\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.749946 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvc5\" (UniqueName: \"kubernetes.io/projected/cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa-kube-api-access-mgvc5\") pod \"ingress-canary-dn4vj\" (UID: \"cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa\") " pod="openshift-ingress-canary/ingress-canary-dn4vj"
Sep 16 19:39:47.749946 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kks92\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-kube-api-access-kks92\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.749946 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42e50fb-1452-4a79-bf3d-ccaf42264113-config-volume\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.749946 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-trusted-ca\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.749946 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7j5\" (UniqueName: \"kubernetes.io/projected/9fd2049f-99b3-41f0-9abf-c2a89df095f3-kube-api-access-xf7j5\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.749946 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.750147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-image-registry-private-configuration\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.750147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.749973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.750147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.750000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42e50fb-1452-4a79-bf3d-ccaf42264113-tmp-dir\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.750147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.750026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d0af4d-0b59-4197-9336-70da10e0f432-ca-trust-extracted\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.750147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.750079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-ca\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.750147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.750101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.750147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.750128    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-certificates\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.750344 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.750153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd2049f-99b3-41f0-9abf-c2a89df095f3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.750344 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.750182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-installation-pull-secrets\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.787352 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.787320    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z26db_07142bb7-f90d-4bb3-b051-671f06731298/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:47.851003 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.850932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-ca\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.851003 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.850962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.851003 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.850992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-certificates\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd2049f-99b3-41f0-9abf-c2a89df095f3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-installation-pull-secrets\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed7aea78-8e03-41ef-b027-ef55194572c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d56d5995d-8447t\" (UID: \"ed7aea78-8e03-41ef-b027-ef55194572c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa-cert\") pod \"ingress-canary-dn4vj\" (UID: \"cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa\") " pod="openshift-ingress-canary/ingress-canary-dn4vj"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-hub\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-tmp\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mt6k\" (UniqueName: \"kubernetes.io/projected/c42e50fb-1452-4a79-bf3d-ccaf42264113-kube-api-access-7mt6k\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.851237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-bound-sa-token\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zwf\" (UniqueName: \"kubernetes.io/projected/ed7aea78-8e03-41ef-b027-ef55194572c1-kube-api-access-z6zwf\") pod \"managed-serviceaccount-addon-agent-d56d5995d-8447t\" (UID: \"ed7aea78-8e03-41ef-b027-ef55194572c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-klusterlet-config\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqxg\" (UniqueName: \"kubernetes.io/projected/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-kube-api-access-jbqxg\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvc5\" (UniqueName: \"kubernetes.io/projected/cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa-kube-api-access-mgvc5\") pod \"ingress-canary-dn4vj\" (UID: \"cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa\") " pod="openshift-ingress-canary/ingress-canary-dn4vj"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kks92\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-kube-api-access-kks92\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42e50fb-1452-4a79-bf3d-ccaf42264113-config-volume\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-trusted-ca\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7j5\" (UniqueName: \"kubernetes.io/projected/9fd2049f-99b3-41f0-9abf-c2a89df095f3-kube-api-access-xf7j5\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-image-registry-private-configuration\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.851685 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.852219 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42e50fb-1452-4a79-bf3d-ccaf42264113-tmp-dir\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.852219 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.851732    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d0af4d-0b59-4197-9336-70da10e0f432-ca-trust-extracted\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.852219 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.852043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd2049f-99b3-41f0-9abf-c2a89df095f3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.852219 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.852090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d0af4d-0b59-4197-9336-70da10e0f432-ca-trust-extracted\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.852418 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.852370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-certificates\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.852680 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.852499    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:39:47.852680 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.852574    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls podName:c42e50fb-1452-4a79-bf3d-ccaf42264113 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:48.352557719 +0000 UTC m=+35.124428111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls") pod "dns-default-ndpfg" (UID: "c42e50fb-1452-4a79-bf3d-ccaf42264113") : secret "dns-default-metrics-tls" not found
Sep 16 19:39:47.852895 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.852852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42e50fb-1452-4a79-bf3d-ccaf42264113-config-volume\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.853616 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.853588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-trusted-ca\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.854393 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.854355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-tmp\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.854690 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.854668    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:39:47.854767 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.854694    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7dbbbf-8k7x9: secret "image-registry-tls" not found
Sep 16 19:39:47.854767 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:47.854744    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls podName:f8d0af4d-0b59-4197-9336-70da10e0f432 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:48.354725956 +0000 UTC m=+35.126596363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls") pod "image-registry-869d7dbbbf-8k7x9" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432") : secret "image-registry-tls" not found
Sep 16 19:39:47.855020 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.854998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42e50fb-1452-4a79-bf3d-ccaf42264113-tmp-dir\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.856465 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.856437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-installation-pull-secrets\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.856576 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.856468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-ca\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.856576 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.856564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed7aea78-8e03-41ef-b027-ef55194572c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d56d5995d-8447t\" (UID: \"ed7aea78-8e03-41ef-b027-ef55194572c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:39:47.856686 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.856605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-hub\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.856781 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.856756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.857010 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.856991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9fd2049f-99b3-41f0-9abf-c2a89df095f3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.857104 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.857083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-image-registry-private-configuration\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.857512 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.857493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-klusterlet-config\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.858556 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.858538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa-cert\") pod \"ingress-canary-dn4vj\" (UID: \"cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa\") " pod="openshift-ingress-canary/ingress-canary-dn4vj"
Sep 16 19:39:47.861578 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.861555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvc5\" (UniqueName: \"kubernetes.io/projected/cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa-kube-api-access-mgvc5\") pod \"ingress-canary-dn4vj\" (UID: \"cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa\") " pod="openshift-ingress-canary/ingress-canary-dn4vj"
Sep 16 19:39:47.861817 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.861797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kks92\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-kube-api-access-kks92\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.863140 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.863116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mt6k\" (UniqueName: \"kubernetes.io/projected/c42e50fb-1452-4a79-bf3d-ccaf42264113-kube-api-access-7mt6k\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:47.863345 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.863321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-bound-sa-token\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:47.863716 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.863701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7j5\" (UniqueName: \"kubernetes.io/projected/9fd2049f-99b3-41f0-9abf-c2a89df095f3-kube-api-access-xf7j5\") pod \"cluster-proxy-proxy-agent-5c9759f857-pdgd2\" (UID: \"9fd2049f-99b3-41f0-9abf-c2a89df095f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.865305 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.865287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqxg\" (UniqueName: \"kubernetes.io/projected/d0b9dda8-9ed0-430b-9f56-6b653f77dc75-kube-api-access-jbqxg\") pod \"klusterlet-addon-workmgr-c7bf964f-jhml9\" (UID: \"d0b9dda8-9ed0-430b-9f56-6b653f77dc75\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.866066 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.866045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zwf\" (UniqueName: \"kubernetes.io/projected/ed7aea78-8e03-41ef-b027-ef55194572c1-kube-api-access-z6zwf\") pod \"managed-serviceaccount-addon-agent-d56d5995d-8447t\" (UID: \"ed7aea78-8e03-41ef-b027-ef55194572c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:39:47.873693 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.873668    2561 generic.go:356] "Generic (PLEG): container finished" podID="15ecb2f2-b01c-4588-b82a-a8f6bde8d89c" containerID="75bc1961ddfcbd3fe97be2116472e98c545947b034a22f4a4d02a856349c4eb7" exitCode=0
Sep 16 19:39:47.873788 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.873711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerDied","Data":"75bc1961ddfcbd3fe97be2116472e98c545947b034a22f4a4d02a856349c4eb7"}
Sep 16 19:39:47.881648 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.881624    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z26db_07142bb7-f90d-4bb3-b051-671f06731298/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:47.931136 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.931109    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:39:47.939226 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.939208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:39:47.944580 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.944565    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:47.968438 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:47.968415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn4vj"
Sep 16 19:39:48.246862 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.246830    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"]
Sep 16 19:39:48.251226 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.251101    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"]
Sep 16 19:39:48.279728 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.279697    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"]
Sep 16 19:39:48.283781 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:48.283750    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b9dda8_9ed0_430b_9f56_6b653f77dc75.slice/crio-fae9d95e204cfe6c2897de385bc7f466d3a08132b676f94a7c1f1b7c719bfe3a WatchSource:0}: Error finding container fae9d95e204cfe6c2897de385bc7f466d3a08132b676f94a7c1f1b7c719bfe3a: Status 404 returned error can't find the container with id fae9d95e204cfe6c2897de385bc7f466d3a08132b676f94a7c1f1b7c719bfe3a
Sep 16 19:39:48.284477 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:48.284436    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fd2049f_99b3_41f0_9abf_c2a89df095f3.slice/crio-186d097020318b3c1844f31df78a383053f8a007c2b6dfa5f7fddbb7e3ecc531 WatchSource:0}: Error finding container 186d097020318b3c1844f31df78a383053f8a007c2b6dfa5f7fddbb7e3ecc531: Status 404 returned error can't find the container with id 186d097020318b3c1844f31df78a383053f8a007c2b6dfa5f7fddbb7e3ecc531
Sep 16 19:39:48.285255 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:48.285226    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded7aea78_8e03_41ef_b027_ef55194572c1.slice/crio-d76853efdc3fd3c191c2415bf09103e1ea7fd842de928627ea9b81df507aa7ff WatchSource:0}: Error finding container d76853efdc3fd3c191c2415bf09103e1ea7fd842de928627ea9b81df507aa7ff: Status 404 returned error can't find the container with id d76853efdc3fd3c191c2415bf09103e1ea7fd842de928627ea9b81df507aa7ff
Sep 16 19:39:48.355555 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.355516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:48.356636 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.355571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:48.356636 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:48.355753    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:39:48.356636 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:48.355776    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7dbbbf-8k7x9: secret "image-registry-tls" not found
Sep 16 19:39:48.356636 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:48.355859    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls podName:f8d0af4d-0b59-4197-9336-70da10e0f432 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:49.355837604 +0000 UTC m=+36.127708013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls") pod "image-registry-869d7dbbbf-8k7x9" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432") : secret "image-registry-tls" not found
Sep 16 19:39:48.356636 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:48.356422    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:39:48.356636 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:48.356490    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls podName:c42e50fb-1452-4a79-bf3d-ccaf42264113 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:49.356474033 +0000 UTC m=+36.128344442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls") pod "dns-default-ndpfg" (UID: "c42e50fb-1452-4a79-bf3d-ccaf42264113") : secret "dns-default-metrics-tls" not found
Sep 16 19:39:48.547706 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.547675    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dn4vj"]
Sep 16 19:39:48.550654 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:39:48.550625    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc075fa1_4b85_4a98_ab9b_2c0a62d4bfaa.slice/crio-7a48416fbef01edc4331a53d672cebd58facef2a2176e5b88b72145f58e52235 WatchSource:0}: Error finding container 7a48416fbef01edc4331a53d672cebd58facef2a2176e5b88b72145f58e52235: Status 404 returned error can't find the container with id 7a48416fbef01edc4331a53d672cebd58facef2a2176e5b88b72145f58e52235
Sep 16 19:39:48.877842 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.877761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9" event={"ID":"d0b9dda8-9ed0-430b-9f56-6b653f77dc75","Type":"ContainerStarted","Data":"fae9d95e204cfe6c2897de385bc7f466d3a08132b676f94a7c1f1b7c719bfe3a"}
Sep 16 19:39:48.880877 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.880850    2561 generic.go:356] "Generic (PLEG): container finished" podID="15ecb2f2-b01c-4588-b82a-a8f6bde8d89c" containerID="410d8d4f4b54ae109190747e9ed196545f709acf9b039bcb615e8aca41ce44d6" exitCode=0
Sep 16 19:39:48.881005 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.880936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerDied","Data":"410d8d4f4b54ae109190747e9ed196545f709acf9b039bcb615e8aca41ce44d6"}
Sep 16 19:39:48.882108 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.882084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t" event={"ID":"ed7aea78-8e03-41ef-b027-ef55194572c1","Type":"ContainerStarted","Data":"d76853efdc3fd3c191c2415bf09103e1ea7fd842de928627ea9b81df507aa7ff"}
Sep 16 19:39:48.883104 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.883086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" event={"ID":"9fd2049f-99b3-41f0-9abf-c2a89df095f3","Type":"ContainerStarted","Data":"186d097020318b3c1844f31df78a383053f8a007c2b6dfa5f7fddbb7e3ecc531"}
Sep 16 19:39:48.884246 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:48.884227    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dn4vj" event={"ID":"cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa","Type":"ContainerStarted","Data":"7a48416fbef01edc4331a53d672cebd58facef2a2176e5b88b72145f58e52235"}
Sep 16 19:39:49.363961 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:49.363717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:49.364710 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:49.363978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:49.364710 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:49.364147    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:39:49.364710 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:49.364163    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7dbbbf-8k7x9: secret "image-registry-tls" not found
Sep 16 19:39:49.364710 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:49.364220    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls podName:f8d0af4d-0b59-4197-9336-70da10e0f432 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:51.364201668 +0000 UTC m=+38.136072060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls") pod "image-registry-869d7dbbbf-8k7x9" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432") : secret "image-registry-tls" not found
Sep 16 19:39:49.364710 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:49.364629    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:39:49.364710 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:49.364680    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls podName:c42e50fb-1452-4a79-bf3d-ccaf42264113 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:51.364665513 +0000 UTC m=+38.136535905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls") pod "dns-default-ndpfg" (UID: "c42e50fb-1452-4a79-bf3d-ccaf42264113") : secret "dns-default-metrics-tls" not found
Sep 16 19:39:49.671909 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:49.671717    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z26db_07142bb7-f90d-4bb3-b051-671f06731298/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:49.732169 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:49.732137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z26db_07142bb7-f90d-4bb3-b051-671f06731298/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:49.915868 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:49.915829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-525g9" event={"ID":"15ecb2f2-b01c-4588-b82a-a8f6bde8d89c","Type":"ContainerStarted","Data":"80d82d54dc6a83d32e4b286edd1d858db93fbd23cea8aa695e904b39defff46f"}
Sep 16 19:39:49.943975 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:49.943877    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-525g9" podStartSLOduration=8.407731887 podStartE2EDuration="35.943855936s" totalImagesPullingTime="27.536124049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:49.9430034 +0000 UTC m=+36.714873812" watchObservedRunningTime="2026-09-16 19:39:49.943855936 +0000 UTC m=+36.715726348"
Sep 16 19:39:50.074528 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:50.074473    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z26db_07142bb7-f90d-4bb3-b051-671f06731298/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:50.173680 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:50.173620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z26db_07142bb7-f90d-4bb3-b051-671f06731298/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:51.384736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:51.384416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:51.384736 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:51.384466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:51.384736 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:51.384595    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:39:51.384736 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:51.384617    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7dbbbf-8k7x9: secret "image-registry-tls" not found
Sep 16 19:39:51.384736 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:51.384679    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls podName:f8d0af4d-0b59-4197-9336-70da10e0f432 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:55.384659785 +0000 UTC m=+42.156530179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls") pod "image-registry-869d7dbbbf-8k7x9" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432") : secret "image-registry-tls" not found
Sep 16 19:39:51.384736 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:51.384594    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:39:51.384736 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:51.384712    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls podName:c42e50fb-1452-4a79-bf3d-ccaf42264113 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:55.384703911 +0000 UTC m=+42.156574301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls") pod "dns-default-ndpfg" (UID: "c42e50fb-1452-4a79-bf3d-ccaf42264113") : secret "dns-default-metrics-tls" not found
Sep 16 19:39:55.419643 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:55.419602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:39:55.419643 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:55.419645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:39:55.420050 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:55.419748    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:39:55.420050 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:55.419753    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:39:55.420050 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:55.419758    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7dbbbf-8k7x9: secret "image-registry-tls" not found
Sep 16 19:39:55.420050 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:55.419828    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls podName:c42e50fb-1452-4a79-bf3d-ccaf42264113 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:03.419810271 +0000 UTC m=+50.191680679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls") pod "dns-default-ndpfg" (UID: "c42e50fb-1452-4a79-bf3d-ccaf42264113") : secret "dns-default-metrics-tls" not found
Sep 16 19:39:55.420050 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:39:55.419853    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls podName:f8d0af4d-0b59-4197-9336-70da10e0f432 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:03.419840928 +0000 UTC m=+50.191711325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls") pod "image-registry-869d7dbbbf-8k7x9" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432") : secret "image-registry-tls" not found
Sep 16 19:39:56.940506 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:56.940442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t" event={"ID":"ed7aea78-8e03-41ef-b027-ef55194572c1","Type":"ContainerStarted","Data":"c6235fc8ebdcfaa3d0a5fffb3a6294a4858e4f669cf3d39e8a393762981c8434"}
Sep 16 19:39:56.942451 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:56.942418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" event={"ID":"9fd2049f-99b3-41f0-9abf-c2a89df095f3","Type":"ContainerStarted","Data":"ab4ee8e083c8724cdd3e37b1b3098ac7bcd5105fd0be781f09464ababa9fce69"}
Sep 16 19:39:56.944119 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:56.944094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dn4vj" event={"ID":"cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa","Type":"ContainerStarted","Data":"99e5bbeb1e5f0f8d79189c877ecdbc7fd8009e4ccd505cab1a2b96d7005c3b9d"}
Sep 16 19:39:56.973469 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:56.973367    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t" podStartSLOduration=30.53916521 podStartE2EDuration="38.973351986s" totalImagesPullingTime="8.434186776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:56.972308703 +0000 UTC m=+43.744179114" watchObservedRunningTime="2026-09-16 19:39:56.973351986 +0000 UTC m=+43.745222397"
Sep 16 19:39:56.989641 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:56.989589    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dn4vj" podStartSLOduration=1.818743838 podStartE2EDuration="9.989570052s" totalImagesPullingTime="8.170826214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:56.988805461 +0000 UTC m=+43.760675907" watchObservedRunningTime="2026-09-16 19:39:56.989570052 +0000 UTC m=+43.761440464"
Sep 16 19:39:57.948894 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:57.948860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9" event={"ID":"d0b9dda8-9ed0-430b-9f56-6b653f77dc75","Type":"ContainerStarted","Data":"4b24c9c94c89e169f0f4f71e23783aafcaaec6930be66790dd502a8439994e95"}
Sep 16 19:39:57.949332 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:57.949031    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:57.951694 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:57.951669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" event={"ID":"9fd2049f-99b3-41f0-9abf-c2a89df095f3","Type":"ContainerStarted","Data":"f4d9a372e029d250fecd71693a8ac58d4a73ba18b440a9405bacd2f6922bf276"}
Sep 16 19:39:57.951932 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:57.951913    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:39:57.952037 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:57.951948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" event={"ID":"9fd2049f-99b3-41f0-9abf-c2a89df095f3","Type":"ContainerStarted","Data":"4937c667d22ce36df28a43970aac0e37f7625d7cecf24d7cc13d4f56befa310e"}
Sep 16 19:39:57.970624 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:57.970577    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9" podStartSLOduration=31.13286639 podStartE2EDuration="39.9705629s" totalImagesPullingTime="8.83769651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:57.969980208 +0000 UTC m=+44.741850623" watchObservedRunningTime="2026-09-16 19:39:57.9705629 +0000 UTC m=+44.742433310"
Sep 16 19:39:58.007953 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:39:58.007909    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" podStartSLOduration=31.558451706 podStartE2EDuration="40.007896047s" totalImagesPullingTime="8.449444341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:58.007178615 +0000 UTC m=+44.779049018" watchObservedRunningTime="2026-09-16 19:39:58.007896047 +0000 UTC m=+44.779766458"
Sep 16 19:40:03.486732 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:03.486693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:40:03.486732 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:03.486733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:40:03.487160 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:03.486853    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:40:03.487160 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:03.486866    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7dbbbf-8k7x9: secret "image-registry-tls" not found
Sep 16 19:40:03.487160 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:03.486909    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls podName:f8d0af4d-0b59-4197-9336-70da10e0f432 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:19.486896642 +0000 UTC m=+66.258767032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls") pod "image-registry-869d7dbbbf-8k7x9" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432") : secret "image-registry-tls" not found
Sep 16 19:40:03.487160 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:03.486853    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:03.487160 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:03.487002    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls podName:c42e50fb-1452-4a79-bf3d-ccaf42264113 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:19.486986514 +0000 UTC m=+66.258856902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls") pod "dns-default-ndpfg" (UID: "c42e50fb-1452-4a79-bf3d-ccaf42264113") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:13.871853 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:13.871822    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhw67"
Sep 16 19:40:19.401818 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.401772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:40:19.404865 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.404846    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 19:40:19.412220 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:19.412203    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 19:40:19.412271 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:19.412261    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:41:23.412246758 +0000 UTC m=+130.184117159 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : secret "metrics-daemon-secret" not found
Sep 16 19:40:19.503007 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.502965    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:40:19.503007 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.503007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:40:19.503192 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.503045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:40:19.503192 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.503072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:40:19.503192 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:19.503111    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:19.503192 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:19.503179    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:40:19.503308 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:19.503192    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls podName:c42e50fb-1452-4a79-bf3d-ccaf42264113 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:51.503172732 +0000 UTC m=+98.275043136 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls") pod "dns-default-ndpfg" (UID: "c42e50fb-1452-4a79-bf3d-ccaf42264113") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:19.503308 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:19.503196    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7dbbbf-8k7x9: secret "image-registry-tls" not found
Sep 16 19:40:19.503308 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:19.503256    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls podName:f8d0af4d-0b59-4197-9336-70da10e0f432 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:51.50324022 +0000 UTC m=+98.275110625 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls") pod "image-registry-869d7dbbbf-8k7x9" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432") : secret "image-registry-tls" not found
Sep 16 19:40:19.506092 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.506075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 19:40:19.506157 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.506075    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 19:40:19.516479 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.516460    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 19:40:19.517441 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.517412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02086a7b-8320-4264-96c3-fba73847dbce-original-pull-secret\") pod \"global-pull-secret-syncer-wct9n\" (UID: \"02086a7b-8320-4264-96c3-fba73847dbce\") " pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:40:19.526925 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.526903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q464f\" (UniqueName: \"kubernetes.io/projected/a742ff26-e9a8-4033-b661-c370dff3279f-kube-api-access-q464f\") pod \"network-check-target-m5zz2\" (UID: \"a742ff26-e9a8-4033-b661-c370dff3279f\") " pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:40:19.787492 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.787463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wct9n"
Sep 16 19:40:19.801138 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.801099    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2djhj\""
Sep 16 19:40:19.808614 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:19.808585    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:40:20.056982 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:20.056906    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wct9n"]
Sep 16 19:40:20.060185 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:40:20.060162    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02086a7b_8320_4264_96c3_fba73847dbce.slice/crio-96e103089f95b5b57bd3181b9b80a25643e05bb1124e1bfa8c8e3b78dbbc35f4 WatchSource:0}: Error finding container 96e103089f95b5b57bd3181b9b80a25643e05bb1124e1bfa8c8e3b78dbbc35f4: Status 404 returned error can't find the container with id 96e103089f95b5b57bd3181b9b80a25643e05bb1124e1bfa8c8e3b78dbbc35f4
Sep 16 19:40:20.082473 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:20.082440    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m5zz2"]
Sep 16 19:40:20.084710 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:40:20.084691    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda742ff26_e9a8_4033_b661_c370dff3279f.slice/crio-3932197e03437e6816afd834c81c5f011a8238d503d655f6a626e5f345a9d14b WatchSource:0}: Error finding container 3932197e03437e6816afd834c81c5f011a8238d503d655f6a626e5f345a9d14b: Status 404 returned error can't find the container with id 3932197e03437e6816afd834c81c5f011a8238d503d655f6a626e5f345a9d14b
Sep 16 19:40:21.019157 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:21.019069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m5zz2" event={"ID":"a742ff26-e9a8-4033-b661-c370dff3279f","Type":"ContainerStarted","Data":"3932197e03437e6816afd834c81c5f011a8238d503d655f6a626e5f345a9d14b"}
Sep 16 19:40:21.020619 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:21.020583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wct9n" event={"ID":"02086a7b-8320-4264-96c3-fba73847dbce","Type":"ContainerStarted","Data":"96e103089f95b5b57bd3181b9b80a25643e05bb1124e1bfa8c8e3b78dbbc35f4"}
Sep 16 19:40:25.039935 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:25.039899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m5zz2" event={"ID":"a742ff26-e9a8-4033-b661-c370dff3279f","Type":"ContainerStarted","Data":"1be758e8988025c9a094c1a9c8f9b4b5f4de5bcb92e937b5dd08143274104820"}
Sep 16 19:40:25.040394 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:25.040014    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:40:25.041553 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:25.041532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wct9n" event={"ID":"02086a7b-8320-4264-96c3-fba73847dbce","Type":"ContainerStarted","Data":"d5c047c0066f30f9bc1f66b10827f0f810f52105d4e2844ee3fb701f37cf47c7"}
Sep 16 19:40:25.091063 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:25.091015    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m5zz2" podStartSLOduration=67.820588846 podStartE2EDuration="1m12.090999212s" totalImagesPullingTime="4.270410366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:25.068856732 +0000 UTC m=+71.840727144" watchObservedRunningTime="2026-09-16 19:40:25.090999212 +0000 UTC m=+71.862869622"
Sep 16 19:40:25.091422 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:25.091398    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wct9n" podStartSLOduration=66.789548881 podStartE2EDuration="1m11.091392628s" totalImagesPullingTime="4.301843747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:25.089880382 +0000 UTC m=+71.861750792" watchObservedRunningTime="2026-09-16 19:40:25.091392628 +0000 UTC m=+71.863263031"
Sep 16 19:40:51.539790 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:51.539738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:40:51.539790 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:51.539790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:40:51.540215 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:51.539889    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:40:51.540215 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:51.539891    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:51.540215 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:51.539966    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls podName:c42e50fb-1452-4a79-bf3d-ccaf42264113 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:55.539950555 +0000 UTC m=+162.311820949 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls") pod "dns-default-ndpfg" (UID: "c42e50fb-1452-4a79-bf3d-ccaf42264113") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:51.540215 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:51.539901    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7dbbbf-8k7x9: secret "image-registry-tls" not found
Sep 16 19:40:51.540215 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:40:51.540034    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls podName:f8d0af4d-0b59-4197-9336-70da10e0f432 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:55.540020212 +0000 UTC m=+162.311890621 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls") pod "image-registry-869d7dbbbf-8k7x9" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432") : secret "image-registry-tls" not found
Sep 16 19:40:56.047328 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:40:56.047296    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m5zz2"
Sep 16 19:41:23.466847 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:23.466805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:41:23.467352 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:23.466955    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 19:41:23.467352 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:23.467020    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs podName:ca3b10ac-d514-413b-aa81-23fbe19df3bd nodeName:}" failed. No retries permitted until 2026-09-16 19:43:25.467005301 +0000 UTC m=+252.238875704 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs") pod "network-metrics-daemon-cc5h9" (UID: "ca3b10ac-d514-413b-aa81-23fbe19df3bd") : secret "metrics-daemon-secret" not found
Sep 16 19:41:43.269013 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.268981    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dnzbm"]
Sep 16 19:41:43.272030 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.272008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.275481 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.275459    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 19:41:43.275731 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.275712    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bsdbz\""
Sep 16 19:41:43.276402 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.276371    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 19:41:43.276470 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.276437    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 19:41:43.276703 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.276687    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 19:41:43.302762 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.302733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.302904 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.302787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f517ca1a-8142-48df-83f2-f442fcd61215-crio-socket\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.302971 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.302893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.302971 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.302941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctw8\" (UniqueName: \"kubernetes.io/projected/f517ca1a-8142-48df-83f2-f442fcd61215-kube-api-access-dctw8\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.303071 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.303001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f517ca1a-8142-48df-83f2-f442fcd61215-data-volume\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.303714 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.303692    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnzbm"]
Sep 16 19:41:43.404282 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.404241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctw8\" (UniqueName: \"kubernetes.io/projected/f517ca1a-8142-48df-83f2-f442fcd61215-kube-api-access-dctw8\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.404282 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.404282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f517ca1a-8142-48df-83f2-f442fcd61215-data-volume\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.404587 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.404409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.404587 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.404492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f517ca1a-8142-48df-83f2-f442fcd61215-crio-socket\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.404587 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.404540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.404724 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:43.404597    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.404724 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.404614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f517ca1a-8142-48df-83f2-f442fcd61215-crio-socket\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.404724 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:43.404640    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.404724 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:43.404672    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:43.90465375 +0000 UTC m=+150.676524152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.404724 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:43.404686    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:43.90468032 +0000 UTC m=+150.676550709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.404906 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.404738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f517ca1a-8142-48df-83f2-f442fcd61215-data-volume\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.416824 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.416803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctw8\" (UniqueName: \"kubernetes.io/projected/f517ca1a-8142-48df-83f2-f442fcd61215-kube-api-access-dctw8\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.908798 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.908666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.908798 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:43.908751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:43.909019 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:43.908874    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.909019 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:43.908905    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.909019 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:43.908939    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:44.908925317 +0000 UTC m=+151.680795719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.909019 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:43.908963    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:44.908945808 +0000 UTC m=+151.680816197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:44.914943 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:44.914905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:44.915470 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:44.915004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:44.915470 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:44.915055    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:44.915470 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:44.915130    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:44.915470 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:44.915144    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:46.915123948 +0000 UTC m=+153.686994357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:44.915470 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:44.915182    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:46.915170405 +0000 UTC m=+153.687040794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:45.182626 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:45.182556    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fjkh7_bed5bc13-28aa-4988-a9ca-0bc86e03637e/dns-node-resolver/0.log"
Sep 16 19:41:45.969221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:45.969195    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5l96d_7529a9e4-1769-414e-939e-79eae5b17d12/node-ca/0.log"
Sep 16 19:41:46.768746 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:46.768715    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn4vj_cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa/serve-healthcheck-canary/0.log"
Sep 16 19:41:46.930860 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:46.930823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:46.931034 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:46.930911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:46.931034 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:46.930958    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:46.931034 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:46.931020    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:46.931187 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:46.931041    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:50.931018959 +0000 UTC m=+157.702889348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:46.931187 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:46.931060    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:50.93104971 +0000 UTC m=+157.702920119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:50.603083 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:50.603040    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9" podUID="f8d0af4d-0b59-4197-9336-70da10e0f432"
Sep 16 19:41:50.673498 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:50.673449    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ndpfg" podUID="c42e50fb-1452-4a79-bf3d-ccaf42264113"
Sep 16 19:41:50.693128 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:50.693100    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cc5h9" podUID="ca3b10ac-d514-413b-aa81-23fbe19df3bd"
Sep 16 19:41:50.962745 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:50.962637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:50.962745 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:50.962695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:50.962943 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:50.962767    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:50.962943 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:50.962790    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:50.962943 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:50.962817    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:58.962805005 +0000 UTC m=+165.734675394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:50.962943 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:41:50.962841    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls podName:f517ca1a-8142-48df-83f2-f442fcd61215 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:58.962828424 +0000 UTC m=+165.734698840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnzbm" (UID: "f517ca1a-8142-48df-83f2-f442fcd61215") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:51.296241 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:51.296208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:41:55.597652 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:55.597598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:41:55.598055 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:55.597705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:41:55.599954 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:55.599928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42e50fb-1452-4a79-bf3d-ccaf42264113-metrics-tls\") pod \"dns-default-ndpfg\" (UID: \"c42e50fb-1452-4a79-bf3d-ccaf42264113\") " pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:41:55.600059 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:55.599962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"image-registry-869d7dbbbf-8k7x9\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") " pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:41:55.800387 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:55.800352    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dtnsg\""
Sep 16 19:41:55.807652 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:55.807627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:41:56.093583 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:56.093551    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869d7dbbbf-8k7x9"]
Sep 16 19:41:56.096811 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:41:56.096782    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d0af4d_0b59_4197_9336_70da10e0f432.slice/crio-7955d539aa9b6091e04d71741a9c92c8cf48e3614146820c13753d1e9cc74435 WatchSource:0}: Error finding container 7955d539aa9b6091e04d71741a9c92c8cf48e3614146820c13753d1e9cc74435: Status 404 returned error can't find the container with id 7955d539aa9b6091e04d71741a9c92c8cf48e3614146820c13753d1e9cc74435
Sep 16 19:41:56.312442 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:56.312404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9" event={"ID":"f8d0af4d-0b59-4197-9336-70da10e0f432","Type":"ContainerStarted","Data":"eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b"}
Sep 16 19:41:56.312636 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:56.312455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9" event={"ID":"f8d0af4d-0b59-4197-9336-70da10e0f432","Type":"ContainerStarted","Data":"7955d539aa9b6091e04d71741a9c92c8cf48e3614146820c13753d1e9cc74435"}
Sep 16 19:41:56.312636 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:56.312498    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:41:56.343903 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:56.343240    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9" podStartSLOduration=162.343218138 podStartE2EDuration="2m42.343218138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:56.342069963 +0000 UTC m=+163.113940373" watchObservedRunningTime="2026-09-16 19:41:56.343218138 +0000 UTC m=+163.115088552"
Sep 16 19:41:57.316604 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.316572    2561 generic.go:356] "Generic (PLEG): container finished" podID="ed7aea78-8e03-41ef-b027-ef55194572c1" containerID="c6235fc8ebdcfaa3d0a5fffb3a6294a4858e4f669cf3d39e8a393762981c8434" exitCode=255
Sep 16 19:41:57.317006 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.316650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t" event={"ID":"ed7aea78-8e03-41ef-b027-ef55194572c1","Type":"ContainerDied","Data":"c6235fc8ebdcfaa3d0a5fffb3a6294a4858e4f669cf3d39e8a393762981c8434"}
Sep 16 19:41:57.317006 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.316974    2561 scope.go:117] "RemoveContainer" containerID="c6235fc8ebdcfaa3d0a5fffb3a6294a4858e4f669cf3d39e8a393762981c8434"
Sep 16 19:41:57.318304 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.318283    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0b9dda8-9ed0-430b-9f56-6b653f77dc75" containerID="4b24c9c94c89e169f0f4f71e23783aafcaaec6930be66790dd502a8439994e95" exitCode=1
Sep 16 19:41:57.318394 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.318325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9" event={"ID":"d0b9dda8-9ed0-430b-9f56-6b653f77dc75","Type":"ContainerDied","Data":"4b24c9c94c89e169f0f4f71e23783aafcaaec6930be66790dd502a8439994e95"}
Sep 16 19:41:57.318779 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.318765    2561 scope.go:117] "RemoveContainer" containerID="4b24c9c94c89e169f0f4f71e23783aafcaaec6930be66790dd502a8439994e95"
Sep 16 19:41:57.940000 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.939954    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t"
Sep 16 19:41:57.945100 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.945079    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:41:57.949756 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:57.949740    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:41:58.323673 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:58.323640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9" event={"ID":"d0b9dda8-9ed0-430b-9f56-6b653f77dc75","Type":"ContainerStarted","Data":"89eb782d19115fa0c5d2c1b9fa82a43cafc2977bd9ee07408a049b61e20d7c9d"}
Sep 16 19:41:58.324092 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:58.323813    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:41:58.324519 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:58.324497    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c7bf964f-jhml9"
Sep 16 19:41:58.325464 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:58.325442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d56d5995d-8447t" event={"ID":"ed7aea78-8e03-41ef-b027-ef55194572c1","Type":"ContainerStarted","Data":"eecb00d8e6120ef16a1b7a9a74b543d696b439bd1dd8aa0a6afc737499abcdcb"}
Sep 16 19:41:59.025498 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:59.025461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:59.025703 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:59.025529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:59.026052 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:59.026028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f517ca1a-8142-48df-83f2-f442fcd61215-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:59.027807 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:59.027784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f517ca1a-8142-48df-83f2-f442fcd61215-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnzbm\" (UID: \"f517ca1a-8142-48df-83f2-f442fcd61215\") " pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:59.180366 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:59.180323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnzbm"
Sep 16 19:41:59.455666 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:41:59.455592    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnzbm"]
Sep 16 19:41:59.459277 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:41:59.459250    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf517ca1a_8142_48df_83f2_f442fcd61215.slice/crio-f5ed32bc4f6f7c8e7dde80e91da94e9d3a372f7c0a18fe1b66b5e1ece0a46880 WatchSource:0}: Error finding container f5ed32bc4f6f7c8e7dde80e91da94e9d3a372f7c0a18fe1b66b5e1ece0a46880: Status 404 returned error can't find the container with id f5ed32bc4f6f7c8e7dde80e91da94e9d3a372f7c0a18fe1b66b5e1ece0a46880
Sep 16 19:42:00.335714 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:00.335671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnzbm" event={"ID":"f517ca1a-8142-48df-83f2-f442fcd61215","Type":"ContainerStarted","Data":"f1b60d24bc551e6edd71bf8d530e9ab394bc992f41e6fc6184540c69fb5075e5"}
Sep 16 19:42:00.335882 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:00.335776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnzbm" event={"ID":"f517ca1a-8142-48df-83f2-f442fcd61215","Type":"ContainerStarted","Data":"5958dedca8c38c64542b61db930bdc3d3676d7b2ec39f41161013bf792c9d4c6"}
Sep 16 19:42:00.335882 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:00.335807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnzbm" event={"ID":"f517ca1a-8142-48df-83f2-f442fcd61215","Type":"ContainerStarted","Data":"f5ed32bc4f6f7c8e7dde80e91da94e9d3a372f7c0a18fe1b66b5e1ece0a46880"}
Sep 16 19:42:02.345129 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:02.345091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnzbm" event={"ID":"f517ca1a-8142-48df-83f2-f442fcd61215","Type":"ContainerStarted","Data":"e7fe9e466f113a58eceaf966c6153a4b712490c90f0385f745db77b97f2fd174"}
Sep 16 19:42:02.366976 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:02.366920    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dnzbm" podStartSLOduration=17.460190361 podStartE2EDuration="19.366905812s" totalImagesPullingTime="1.906715451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:41:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:02.366421022 +0000 UTC m=+169.138291434" watchObservedRunningTime="2026-09-16 19:42:02.366905812 +0000 UTC m=+169.138776223"
Sep 16 19:42:02.677978 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:02.677879    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:42:02.680923 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:02.680898    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdb5m\""
Sep 16 19:42:02.688999 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:02.688970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:42:02.970026 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:02.970003    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ndpfg"]
Sep 16 19:42:02.971808 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:42:02.971779    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42e50fb_1452_4a79_bf3d_ccaf42264113.slice/crio-92e04ec80ffae7994cba6d5bc8c18f25b0b683902e9d9a315fbe5e9db60dbbe8 WatchSource:0}: Error finding container 92e04ec80ffae7994cba6d5bc8c18f25b0b683902e9d9a315fbe5e9db60dbbe8: Status 404 returned error can't find the container with id 92e04ec80ffae7994cba6d5bc8c18f25b0b683902e9d9a315fbe5e9db60dbbe8
Sep 16 19:42:03.349168 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:03.349131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndpfg" event={"ID":"c42e50fb-1452-4a79-bf3d-ccaf42264113","Type":"ContainerStarted","Data":"92e04ec80ffae7994cba6d5bc8c18f25b0b683902e9d9a315fbe5e9db60dbbe8"}
Sep 16 19:42:04.677095 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:04.677071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:42:05.358230 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:05.358194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndpfg" event={"ID":"c42e50fb-1452-4a79-bf3d-ccaf42264113","Type":"ContainerStarted","Data":"16dfb98e9818366e62a8b9f3e0157ff17726e8b2fed77b03862207e9b5c0edd1"}
Sep 16 19:42:05.358432 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:05.358248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndpfg" event={"ID":"c42e50fb-1452-4a79-bf3d-ccaf42264113","Type":"ContainerStarted","Data":"87d03f864314300cd207cbda10ddc1e5ee95b293c49315dfa4c4df16f9d2d456"}
Sep 16 19:42:05.358432 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:05.358341    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:42:15.362898 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:15.362819    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ndpfg"
Sep 16 19:42:15.381693 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:15.381645    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ndpfg" podStartSLOduration=146.762294885 podStartE2EDuration="2m28.381629979s" totalImagesPullingTime="1.619335094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:05.388392152 +0000 UTC m=+172.160262554" watchObservedRunningTime="2026-09-16 19:42:15.381629979 +0000 UTC m=+182.153500391"
Sep 16 19:42:15.812296 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:15.812261    2561 patch_prober.go:28] interesting pod/image-registry-869d7dbbbf-8k7x9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 19:42:15.812475 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:15.812327    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9" podUID="f8d0af4d-0b59-4197-9336-70da10e0f432" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:42:17.323517 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:17.323489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:42:20.976996 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.976961    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lm7d8"]
Sep 16 19:42:20.982816 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.982791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:20.985355 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.985336    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 19:42:20.985491 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.985389    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sl9cc\""
Sep 16 19:42:20.988279 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.988260    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 19:42:20.988495 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.988479    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 19:42:20.988578 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.988507    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 19:42:20.991257 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.991234    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 19:42:20.994553 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:20.994538    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 19:42:21.097660 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.097620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-wtmp\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.097660 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.097657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.097915 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.097700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-root\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.097915 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.097787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-metrics-client-ca\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.097915 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.097821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-tls\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.097915 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.097853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-accelerators-collector-config\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.097915 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.097889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-sys\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.098126 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.097942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4zb\" (UniqueName: \"kubernetes.io/projected/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-kube-api-access-vn4zb\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.098126 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.098009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-textfile\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199263 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-sys\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199263 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4zb\" (UniqueName: \"kubernetes.io/projected/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-kube-api-access-vn4zb\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199433 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-textfile\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199433 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-wtmp\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199433 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199433 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-sys\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199433 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-root\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199433 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-root\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199659 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-metrics-client-ca\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199659 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-tls\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199659 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-wtmp\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199659 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.199560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-accelerators-collector-config\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.199659 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:42:21.199636    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 19:42:21.199864 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:42:21.199702    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-tls podName:67c8c80b-22e6-4698-9a1d-b9f25ca9a089 nodeName:}" failed. No retries permitted until 2026-09-16 19:42:21.699681751 +0000 UTC m=+188.471552154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-tls") pod "node-exporter-lm7d8" (UID: "67c8c80b-22e6-4698-9a1d-b9f25ca9a089") : secret "node-exporter-tls" not found
Sep 16 19:42:21.203883 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.200176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-textfile\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.203883 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.200446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-metrics-client-ca\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.203883 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.200586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-accelerators-collector-config\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.204586 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.204563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.209021 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.209001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4zb\" (UniqueName: \"kubernetes.io/projected/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-kube-api-access-vn4zb\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.704030 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.703991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-tls\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.706438 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.706370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67c8c80b-22e6-4698-9a1d-b9f25ca9a089-node-exporter-tls\") pod \"node-exporter-lm7d8\" (UID: \"67c8c80b-22e6-4698-9a1d-b9f25ca9a089\") " pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.892249 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:21.892211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lm7d8"
Sep 16 19:42:21.904710 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:42:21.904681    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c8c80b_22e6_4698_9a1d_b9f25ca9a089.slice/crio-6bc25390ad7753ed0582df920eee1bccbc9d3ca38462be3c1845fd09a961d886 WatchSource:0}: Error finding container 6bc25390ad7753ed0582df920eee1bccbc9d3ca38462be3c1845fd09a961d886: Status 404 returned error can't find the container with id 6bc25390ad7753ed0582df920eee1bccbc9d3ca38462be3c1845fd09a961d886
Sep 16 19:42:22.413199 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:22.413161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lm7d8" event={"ID":"67c8c80b-22e6-4698-9a1d-b9f25ca9a089","Type":"ContainerStarted","Data":"6bc25390ad7753ed0582df920eee1bccbc9d3ca38462be3c1845fd09a961d886"}
Sep 16 19:42:23.418080 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:23.418039    2561 generic.go:356] "Generic (PLEG): container finished" podID="67c8c80b-22e6-4698-9a1d-b9f25ca9a089" containerID="21e1617ea723b404cd5cc14af3c8112e129879da00719d75c9b8429525afa0a4" exitCode=0
Sep 16 19:42:23.418534 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:23.418116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lm7d8" event={"ID":"67c8c80b-22e6-4698-9a1d-b9f25ca9a089","Type":"ContainerDied","Data":"21e1617ea723b404cd5cc14af3c8112e129879da00719d75c9b8429525afa0a4"}
Sep 16 19:42:24.423290 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:24.423254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lm7d8" event={"ID":"67c8c80b-22e6-4698-9a1d-b9f25ca9a089","Type":"ContainerStarted","Data":"fa410e601cefc8d61e70e2ed80d091b234ffc7a8b59cb6bbbecbace056505f28"}
Sep 16 19:42:24.423697 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:24.423317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lm7d8" event={"ID":"67c8c80b-22e6-4698-9a1d-b9f25ca9a089","Type":"ContainerStarted","Data":"efd0b62bf6537c935ee34439364e4433e592b2ec4df8297c761a522793cc21be"}
Sep 16 19:42:24.445859 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:24.445807    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lm7d8" podStartSLOduration=3.450056602 podStartE2EDuration="4.445792749s" totalImagesPullingTime="933.093572ms" totalInitContainerRuntime="62.642575ms" isStatefulPod=true podCreationTimestamp="2026-09-16 19:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:24.444851157 +0000 UTC m=+191.216721579" watchObservedRunningTime="2026-09-16 19:42:24.445792749 +0000 UTC m=+191.217663158"
Sep 16 19:42:27.933015 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:27.932968    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" podUID="9fd2049f-99b3-41f0-9abf-c2a89df095f3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 19:42:28.707406 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:28.707359    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-869d7dbbbf-8k7x9"]
Sep 16 19:42:37.932408 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:37.932351    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" podUID="9fd2049f-99b3-41f0-9abf-c2a89df095f3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 19:42:47.932797 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:47.932753    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" podUID="9fd2049f-99b3-41f0-9abf-c2a89df095f3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 19:42:47.933266 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:47.932841    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2"
Sep 16 19:42:47.933472 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:47.933435    2561 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f4d9a372e029d250fecd71693a8ac58d4a73ba18b440a9405bacd2f6922bf276"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 19:42:47.933562 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:47.933498    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" podUID="9fd2049f-99b3-41f0-9abf-c2a89df095f3" containerName="service-proxy" containerID="cri-o://f4d9a372e029d250fecd71693a8ac58d4a73ba18b440a9405bacd2f6922bf276" gracePeriod=30
Sep 16 19:42:48.504810 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:48.504772    2561 generic.go:356] "Generic (PLEG): container finished" podID="9fd2049f-99b3-41f0-9abf-c2a89df095f3" containerID="f4d9a372e029d250fecd71693a8ac58d4a73ba18b440a9405bacd2f6922bf276" exitCode=2
Sep 16 19:42:48.504971 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:48.504842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" event={"ID":"9fd2049f-99b3-41f0-9abf-c2a89df095f3","Type":"ContainerDied","Data":"f4d9a372e029d250fecd71693a8ac58d4a73ba18b440a9405bacd2f6922bf276"}
Sep 16 19:42:48.504971 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:48.504893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c9759f857-pdgd2" event={"ID":"9fd2049f-99b3-41f0-9abf-c2a89df095f3","Type":"ContainerStarted","Data":"178cb2c3dd34d1033e3fd612cada64829ae571024563358bb394c3735b9b115f"}
Sep 16 19:42:53.725875 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:53.725817    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9" podUID="f8d0af4d-0b59-4197-9336-70da10e0f432" containerName="registry" containerID="cri-o://eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b" gracePeriod=30
Sep 16 19:42:53.962268 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:53.962243    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:42:54.050803 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.050772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") pod \"f8d0af4d-0b59-4197-9336-70da10e0f432\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") "
Sep 16 19:42:54.050973 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.050809    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-certificates\") pod \"f8d0af4d-0b59-4197-9336-70da10e0f432\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") "
Sep 16 19:42:54.050973 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.050840    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-trusted-ca\") pod \"f8d0af4d-0b59-4197-9336-70da10e0f432\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") "
Sep 16 19:42:54.050973 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.050873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kks92\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-kube-api-access-kks92\") pod \"f8d0af4d-0b59-4197-9336-70da10e0f432\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") "
Sep 16 19:42:54.050973 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.050919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-installation-pull-secrets\") pod \"f8d0af4d-0b59-4197-9336-70da10e0f432\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") "
Sep 16 19:42:54.050973 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.050954    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-image-registry-private-configuration\") pod \"f8d0af4d-0b59-4197-9336-70da10e0f432\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") "
Sep 16 19:42:54.051224 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.051101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-bound-sa-token\") pod \"f8d0af4d-0b59-4197-9336-70da10e0f432\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") "
Sep 16 19:42:54.051224 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.051172    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d0af4d-0b59-4197-9336-70da10e0f432-ca-trust-extracted\") pod \"f8d0af4d-0b59-4197-9336-70da10e0f432\" (UID: \"f8d0af4d-0b59-4197-9336-70da10e0f432\") "
Sep 16 19:42:54.051334 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.051219    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f8d0af4d-0b59-4197-9336-70da10e0f432" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:54.051334 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.051306    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f8d0af4d-0b59-4197-9336-70da10e0f432" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:54.051464 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.051436    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-certificates\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:42:54.051464 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.051458    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d0af4d-0b59-4197-9336-70da10e0f432-trusted-ca\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:42:54.053331 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.053285    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f8d0af4d-0b59-4197-9336-70da10e0f432" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:54.053476 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.053395    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f8d0af4d-0b59-4197-9336-70da10e0f432" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:54.053476 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.053466    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f8d0af4d-0b59-4197-9336-70da10e0f432" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:54.053578 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.053554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-kube-api-access-kks92" (OuterVolumeSpecName: "kube-api-access-kks92") pod "f8d0af4d-0b59-4197-9336-70da10e0f432" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432"). InnerVolumeSpecName "kube-api-access-kks92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:54.053658 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.053640    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f8d0af4d-0b59-4197-9336-70da10e0f432" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:54.059414 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.059353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d0af4d-0b59-4197-9336-70da10e0f432-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f8d0af4d-0b59-4197-9336-70da10e0f432" (UID: "f8d0af4d-0b59-4197-9336-70da10e0f432"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:54.151870 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.151842    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-registry-tls\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:42:54.151870 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.151865    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kks92\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-kube-api-access-kks92\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:42:54.151870 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.151873    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-installation-pull-secrets\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:42:54.152044 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.151882    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d0af4d-0b59-4197-9336-70da10e0f432-image-registry-private-configuration\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:42:54.152044 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.151892    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d0af4d-0b59-4197-9336-70da10e0f432-bound-sa-token\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:42:54.152044 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.151900    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d0af4d-0b59-4197-9336-70da10e0f432-ca-trust-extracted\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:42:54.524666 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.524628    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8d0af4d-0b59-4197-9336-70da10e0f432" containerID="eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b" exitCode=0
Sep 16 19:42:54.524826 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.524678    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9"
Sep 16 19:42:54.524826 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.524683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9" event={"ID":"f8d0af4d-0b59-4197-9336-70da10e0f432","Type":"ContainerDied","Data":"eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b"}
Sep 16 19:42:54.524826 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.524732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869d7dbbbf-8k7x9" event={"ID":"f8d0af4d-0b59-4197-9336-70da10e0f432","Type":"ContainerDied","Data":"7955d539aa9b6091e04d71741a9c92c8cf48e3614146820c13753d1e9cc74435"}
Sep 16 19:42:54.524826 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.524760    2561 scope.go:117] "RemoveContainer" containerID="eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b"
Sep 16 19:42:54.535919 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.535894    2561 scope.go:117] "RemoveContainer" containerID="eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b"
Sep 16 19:42:54.536208 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:42:54.536186    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b\": container with ID starting with eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b not found: ID does not exist" containerID="eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b"
Sep 16 19:42:54.536279 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.536215    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b"} err="failed to get container status \"eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b\": rpc error: code = NotFound desc = could not find container \"eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b\": container with ID starting with eee2ebb3a1b46aab7bb8d7dab5c18c496ed1dd3f6b460d5aac9b6a4868a5494b not found: ID does not exist"
Sep 16 19:42:54.549129 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.549105    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-869d7dbbbf-8k7x9"]
Sep 16 19:42:54.553010 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:54.552992    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-869d7dbbbf-8k7x9"]
Sep 16 19:42:55.682060 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:42:55.682025    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d0af4d-0b59-4197-9336-70da10e0f432" path="/var/lib/kubelet/pods/f8d0af4d-0b59-4197-9336-70da10e0f432/volumes"
Sep 16 19:43:25.471796 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:25.471756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:43:25.474011 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:25.473989    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3b10ac-d514-413b-aa81-23fbe19df3bd-metrics-certs\") pod \"network-metrics-daemon-cc5h9\" (UID: \"ca3b10ac-d514-413b-aa81-23fbe19df3bd\") " pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:43:25.680612 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:25.680580    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmvpx\""
Sep 16 19:43:25.688158 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:25.688137    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc5h9"
Sep 16 19:43:25.963427 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:25.963388    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cc5h9"]
Sep 16 19:43:25.966149 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:43:25.966119    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3b10ac_d514_413b_aa81_23fbe19df3bd.slice/crio-4fea12571a68dd7831ac8741932df1999dcce4fcbb9ab9a89c254967d79afaee WatchSource:0}: Error finding container 4fea12571a68dd7831ac8741932df1999dcce4fcbb9ab9a89c254967d79afaee: Status 404 returned error can't find the container with id 4fea12571a68dd7831ac8741932df1999dcce4fcbb9ab9a89c254967d79afaee
Sep 16 19:43:26.633656 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:26.633618    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cc5h9" event={"ID":"ca3b10ac-d514-413b-aa81-23fbe19df3bd","Type":"ContainerStarted","Data":"4fea12571a68dd7831ac8741932df1999dcce4fcbb9ab9a89c254967d79afaee"}
Sep 16 19:43:27.638597 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:27.638558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cc5h9" event={"ID":"ca3b10ac-d514-413b-aa81-23fbe19df3bd","Type":"ContainerStarted","Data":"48ca6daad51398639ea191cdd62380f7e1f051644162d9c5981a5bbc012bef60"}
Sep 16 19:43:27.638986 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:27.638617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cc5h9" event={"ID":"ca3b10ac-d514-413b-aa81-23fbe19df3bd","Type":"ContainerStarted","Data":"b6317bfb50030856bc2447c5cf81f79212dbc44fad8967a472c6ecaeda396cb2"}
Sep 16 19:43:27.659339 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:43:27.659287    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cc5h9" podStartSLOduration=253.769183689 podStartE2EDuration="4m14.659271523s" totalImagesPullingTime="890.087834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:27.657205871 +0000 UTC m=+254.429076281" watchObservedRunningTime="2026-09-16 19:43:27.659271523 +0000 UTC m=+254.431141934"
Sep 16 19:44:13.593085 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:44:13.593057    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:44:13.598420 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:44:13.598401    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:44:13.598915 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:44:13.598892    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 19:46:23.023906 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.023868    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"]
Sep 16 19:46:23.024360 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.024077    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d0af4d-0b59-4197-9336-70da10e0f432" containerName="registry"
Sep 16 19:46:23.024360 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.024087    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d0af4d-0b59-4197-9336-70da10e0f432" containerName="registry"
Sep 16 19:46:23.024360 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.024137    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d0af4d-0b59-4197-9336-70da10e0f432" containerName="registry"
Sep 16 19:46:23.026798 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.026781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"
Sep 16 19:46:23.032190 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.032169    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:46:23.032347 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.032328    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-k5m46\""
Sep 16 19:46:23.033597 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.033578    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 19:46:23.073482 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.073453    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"]
Sep 16 19:46:23.153261 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.153227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxzm\" (UniqueName: \"kubernetes.io/projected/63c724a5-d7e4-45ca-9383-5daef1874d42-kube-api-access-srxzm\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f7ccv\" (UID: \"63c724a5-d7e4-45ca-9383-5daef1874d42\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"
Sep 16 19:46:23.153427 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.153278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c724a5-d7e4-45ca-9383-5daef1874d42-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f7ccv\" (UID: \"63c724a5-d7e4-45ca-9383-5daef1874d42\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"
Sep 16 19:46:23.254545 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.254512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c724a5-d7e4-45ca-9383-5daef1874d42-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f7ccv\" (UID: \"63c724a5-d7e4-45ca-9383-5daef1874d42\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"
Sep 16 19:46:23.254702 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.254580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxzm\" (UniqueName: \"kubernetes.io/projected/63c724a5-d7e4-45ca-9383-5daef1874d42-kube-api-access-srxzm\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f7ccv\" (UID: \"63c724a5-d7e4-45ca-9383-5daef1874d42\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"
Sep 16 19:46:23.254901 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.254881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c724a5-d7e4-45ca-9383-5daef1874d42-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f7ccv\" (UID: \"63c724a5-d7e4-45ca-9383-5daef1874d42\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"
Sep 16 19:46:23.264312 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.264292    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxzm\" (UniqueName: \"kubernetes.io/projected/63c724a5-d7e4-45ca-9383-5daef1874d42-kube-api-access-srxzm\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f7ccv\" (UID: \"63c724a5-d7e4-45ca-9383-5daef1874d42\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"
Sep 16 19:46:23.335878 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.335786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"
Sep 16 19:46:23.616964 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.616780    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv"]
Sep 16 19:46:23.620072 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:46:23.620036    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c724a5_d7e4_45ca_9383_5daef1874d42.slice/crio-9b5eb032aa33c9cf800f83e3f4861465a1ddf68f7f6c917b62712f28e516aa18 WatchSource:0}: Error finding container 9b5eb032aa33c9cf800f83e3f4861465a1ddf68f7f6c917b62712f28e516aa18: Status 404 returned error can't find the container with id 9b5eb032aa33c9cf800f83e3f4861465a1ddf68f7f6c917b62712f28e516aa18
Sep 16 19:46:23.622620 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:23.622605    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:46:24.210257 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:24.210225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv" event={"ID":"63c724a5-d7e4-45ca-9383-5daef1874d42","Type":"ContainerStarted","Data":"9b5eb032aa33c9cf800f83e3f4861465a1ddf68f7f6c917b62712f28e516aa18"}
Sep 16 19:46:27.224750 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:27.224711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv" event={"ID":"63c724a5-d7e4-45ca-9383-5daef1874d42","Type":"ContainerStarted","Data":"25453715747c9b203ebc23d5ddb3f335ba441e1ae9808c6957ed6bb691f0aa4a"}
Sep 16 19:46:30.090398 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.090319    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f7ccv" podStartSLOduration=5.318408172 podStartE2EDuration="8.090301203s" totalImagesPullingTime="2.771893031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:27.300849147 +0000 UTC m=+434.072719569" watchObservedRunningTime="2026-09-16 19:46:30.090301203 +0000 UTC m=+436.862171614"
Sep 16 19:46:30.090863 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.090846    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"]
Sep 16 19:46:30.093920 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.093905    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"
Sep 16 19:46:30.098010 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.097989    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ff749\""
Sep 16 19:46:30.098276 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.098259    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 19:46:30.099171 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.099157    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 19:46:30.106326 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.106301    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"]
Sep 16 19:46:30.202930 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.202893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-567gb\" (UniqueName: \"kubernetes.io/projected/77c6537e-eba4-49ff-8474-e3a2a5d9935b-kube-api-access-567gb\") pod \"cert-manager-cainjector-559cdd8c87-mmh6d\" (UID: \"77c6537e-eba4-49ff-8474-e3a2a5d9935b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"
Sep 16 19:46:30.202930 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.202937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77c6537e-eba4-49ff-8474-e3a2a5d9935b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mmh6d\" (UID: \"77c6537e-eba4-49ff-8474-e3a2a5d9935b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"
Sep 16 19:46:30.303854 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.303825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77c6537e-eba4-49ff-8474-e3a2a5d9935b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mmh6d\" (UID: \"77c6537e-eba4-49ff-8474-e3a2a5d9935b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"
Sep 16 19:46:30.304033 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.303883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-567gb\" (UniqueName: \"kubernetes.io/projected/77c6537e-eba4-49ff-8474-e3a2a5d9935b-kube-api-access-567gb\") pod \"cert-manager-cainjector-559cdd8c87-mmh6d\" (UID: \"77c6537e-eba4-49ff-8474-e3a2a5d9935b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"
Sep 16 19:46:30.323336 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.323313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-567gb\" (UniqueName: \"kubernetes.io/projected/77c6537e-eba4-49ff-8474-e3a2a5d9935b-kube-api-access-567gb\") pod \"cert-manager-cainjector-559cdd8c87-mmh6d\" (UID: \"77c6537e-eba4-49ff-8474-e3a2a5d9935b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"
Sep 16 19:46:30.325412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.325397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77c6537e-eba4-49ff-8474-e3a2a5d9935b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mmh6d\" (UID: \"77c6537e-eba4-49ff-8474-e3a2a5d9935b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"
Sep 16 19:46:30.402869 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.402789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"
Sep 16 19:46:30.675912 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:30.675821    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d"]
Sep 16 19:46:30.678916 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:46:30.678890    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77c6537e_eba4_49ff_8474_e3a2a5d9935b.slice/crio-3533b97f8651214250d570ece2bfcf85dd1c85369ee1cd723d5abb2525387175 WatchSource:0}: Error finding container 3533b97f8651214250d570ece2bfcf85dd1c85369ee1cd723d5abb2525387175: Status 404 returned error can't find the container with id 3533b97f8651214250d570ece2bfcf85dd1c85369ee1cd723d5abb2525387175
Sep 16 19:46:31.239897 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:31.239859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d" event={"ID":"77c6537e-eba4-49ff-8474-e3a2a5d9935b","Type":"ContainerStarted","Data":"3533b97f8651214250d570ece2bfcf85dd1c85369ee1cd723d5abb2525387175"}
Sep 16 19:46:33.249214 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:33.249179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d" event={"ID":"77c6537e-eba4-49ff-8474-e3a2a5d9935b","Type":"ContainerStarted","Data":"82c266a71797f0a801365396e5b294c71f1df0db6ef16645acc7775c91206324"}
Sep 16 19:46:33.272371 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:33.272320    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mmh6d" podStartSLOduration=0.778923596 podStartE2EDuration="3.272305856s" totalImagesPullingTime="2.49338226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:33.270596705 +0000 UTC m=+440.042467114" watchObservedRunningTime="2026-09-16 19:46:33.272305856 +0000 UTC m=+440.044176267"
Sep 16 19:46:38.801647 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:38.801571    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j98qz"]
Sep 16 19:46:38.804631 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:38.804610    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-j98qz"
Sep 16 19:46:38.807297 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:38.807279    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ndf9t\""
Sep 16 19:46:38.814702 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:38.814677    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j98qz"]
Sep 16 19:46:38.970309 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:38.970251    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnll\" (UniqueName: \"kubernetes.io/projected/e21798ac-5d5d-4f71-9e1c-a42aabf01b5b-kube-api-access-mnnll\") pod \"cert-manager-55b4d954d5-j98qz\" (UID: \"e21798ac-5d5d-4f71-9e1c-a42aabf01b5b\") " pod="cert-manager/cert-manager-55b4d954d5-j98qz"
Sep 16 19:46:38.970309 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:38.970307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e21798ac-5d5d-4f71-9e1c-a42aabf01b5b-bound-sa-token\") pod \"cert-manager-55b4d954d5-j98qz\" (UID: \"e21798ac-5d5d-4f71-9e1c-a42aabf01b5b\") " pod="cert-manager/cert-manager-55b4d954d5-j98qz"
Sep 16 19:46:39.071441 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:39.071330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnll\" (UniqueName: \"kubernetes.io/projected/e21798ac-5d5d-4f71-9e1c-a42aabf01b5b-kube-api-access-mnnll\") pod \"cert-manager-55b4d954d5-j98qz\" (UID: \"e21798ac-5d5d-4f71-9e1c-a42aabf01b5b\") " pod="cert-manager/cert-manager-55b4d954d5-j98qz"
Sep 16 19:46:39.071441 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:39.071397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e21798ac-5d5d-4f71-9e1c-a42aabf01b5b-bound-sa-token\") pod \"cert-manager-55b4d954d5-j98qz\" (UID: \"e21798ac-5d5d-4f71-9e1c-a42aabf01b5b\") " pod="cert-manager/cert-manager-55b4d954d5-j98qz"
Sep 16 19:46:39.080386 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:39.080341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e21798ac-5d5d-4f71-9e1c-a42aabf01b5b-bound-sa-token\") pod \"cert-manager-55b4d954d5-j98qz\" (UID: \"e21798ac-5d5d-4f71-9e1c-a42aabf01b5b\") " pod="cert-manager/cert-manager-55b4d954d5-j98qz"
Sep 16 19:46:39.080589 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:39.080568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnll\" (UniqueName: \"kubernetes.io/projected/e21798ac-5d5d-4f71-9e1c-a42aabf01b5b-kube-api-access-mnnll\") pod \"cert-manager-55b4d954d5-j98qz\" (UID: \"e21798ac-5d5d-4f71-9e1c-a42aabf01b5b\") " pod="cert-manager/cert-manager-55b4d954d5-j98qz"
Sep 16 19:46:39.113362 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:39.113326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-j98qz"
Sep 16 19:46:39.383775 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:39.383696    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j98qz"]
Sep 16 19:46:39.386956 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:46:39.386925    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode21798ac_5d5d_4f71_9e1c_a42aabf01b5b.slice/crio-e6abbb77bcd2af43d66bb70e744c7f3d22ef976a83418cd672d40892a1387453 WatchSource:0}: Error finding container e6abbb77bcd2af43d66bb70e744c7f3d22ef976a83418cd672d40892a1387453: Status 404 returned error can't find the container with id e6abbb77bcd2af43d66bb70e744c7f3d22ef976a83418cd672d40892a1387453
Sep 16 19:46:40.272319 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:40.272284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-j98qz" event={"ID":"e21798ac-5d5d-4f71-9e1c-a42aabf01b5b","Type":"ContainerStarted","Data":"084b17f8d42c5d09ee08fc920892550118eb0bf52898b1cfa77a73da6f4f6c6f"}
Sep 16 19:46:40.272319 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:40.272327    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-j98qz" event={"ID":"e21798ac-5d5d-4f71-9e1c-a42aabf01b5b","Type":"ContainerStarted","Data":"e6abbb77bcd2af43d66bb70e744c7f3d22ef976a83418cd672d40892a1387453"}
Sep 16 19:46:40.291234 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:46:40.291177    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-j98qz" podStartSLOduration=2.291161156 podStartE2EDuration="2.291161156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:40.289314608 +0000 UTC m=+447.061185021" watchObservedRunningTime="2026-09-16 19:46:40.291161156 +0000 UTC m=+447.063031566"
Sep 16 19:47:09.494622 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.494587    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"]
Sep 16 19:47:09.497497 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.497480    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.506915 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.506896    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 19:47:09.506999 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.506896    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 19:47:09.508272 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.508252    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 19:47:09.508393 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.508297    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 19:47:09.508393 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.508347    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rzklv\""
Sep 16 19:47:09.508393 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.508354    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:47:09.518661 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.518638    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"]
Sep 16 19:47:09.687369 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.687337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/965e977c-2a61-49ae-9d6b-0e071efcc307-manager-config\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.687565 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.687401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/965e977c-2a61-49ae-9d6b-0e071efcc307-cert\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.687565 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.687425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/965e977c-2a61-49ae-9d6b-0e071efcc307-metrics-cert\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.687565 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.687471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72r2\" (UniqueName: \"kubernetes.io/projected/965e977c-2a61-49ae-9d6b-0e071efcc307-kube-api-access-z72r2\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.788695 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.788655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/965e977c-2a61-49ae-9d6b-0e071efcc307-manager-config\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.788695 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.788697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/965e977c-2a61-49ae-9d6b-0e071efcc307-cert\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.788949 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.788714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/965e977c-2a61-49ae-9d6b-0e071efcc307-metrics-cert\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.788949 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.788783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72r2\" (UniqueName: \"kubernetes.io/projected/965e977c-2a61-49ae-9d6b-0e071efcc307-kube-api-access-z72r2\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.789451 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.789426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/965e977c-2a61-49ae-9d6b-0e071efcc307-manager-config\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.791334 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.791311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/965e977c-2a61-49ae-9d6b-0e071efcc307-cert\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.791441 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.791317    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/965e977c-2a61-49ae-9d6b-0e071efcc307-metrics-cert\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:09.817556 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:09.817529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72r2\" (UniqueName: \"kubernetes.io/projected/965e977c-2a61-49ae-9d6b-0e071efcc307-kube-api-access-z72r2\") pod \"lws-controller-manager-b4645d7c8-gvj24\" (UID: \"965e977c-2a61-49ae-9d6b-0e071efcc307\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:10.106405 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:10.106302    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:10.383117 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:10.382929    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"]
Sep 16 19:47:10.386089 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:47:10.386059    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965e977c_2a61_49ae_9d6b_0e071efcc307.slice/crio-1f09a7b79d531067d543444969fe351c681ea01ede431f4e72733c755c784048 WatchSource:0}: Error finding container 1f09a7b79d531067d543444969fe351c681ea01ede431f4e72733c755c784048: Status 404 returned error can't find the container with id 1f09a7b79d531067d543444969fe351c681ea01ede431f4e72733c755c784048
Sep 16 19:47:11.379997 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:11.379964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24" event={"ID":"965e977c-2a61-49ae-9d6b-0e071efcc307","Type":"ContainerStarted","Data":"1f09a7b79d531067d543444969fe351c681ea01ede431f4e72733c755c784048"}
Sep 16 19:47:13.388612 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:13.388575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24" event={"ID":"965e977c-2a61-49ae-9d6b-0e071efcc307","Type":"ContainerStarted","Data":"477c4714bd3f4a4a86edcc66bb664b52dbef5c4cd3cf7a1b696ed39fa3dcdd16"}
Sep 16 19:47:13.388978 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:13.388711    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:13.415123 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:13.415082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24" podStartSLOduration=2.386591681 podStartE2EDuration="4.415067219s" totalImagesPullingTime="2.028475538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:13.413703535 +0000 UTC m=+480.185573946" watchObservedRunningTime="2026-09-16 19:47:13.415067219 +0000 UTC m=+480.186937630"
Sep 16 19:47:24.394510 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:24.394478    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-gvj24"
Sep 16 19:47:43.619647 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.619613    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"]
Sep 16 19:47:43.622777 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.622758    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:43.626393 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.626361    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 19:47:43.626492 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.626472    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 19:47:43.626604 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.626586    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tqg27\""
Sep 16 19:47:43.627548 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.627531    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 19:47:43.627677 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.627657    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 19:47:43.634276 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.634251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"]
Sep 16 19:47:43.733560 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.733527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:43.733560 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.733564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8scgb\" (UniqueName: \"kubernetes.io/projected/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-kube-api-access-8scgb\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:43.733799 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.733662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:43.834710 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.834677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:43.834710 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.834715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8scgb\" (UniqueName: \"kubernetes.io/projected/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-kube-api-access-8scgb\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:43.834923 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.834768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:43.834923 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:47:43.834810    2561 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 19:47:43.834923 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:47:43.834872    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-plugin-serving-cert podName:fa4d18f8-2eb3-4ad0-9448-7b601d74aeab nodeName:}" failed. No retries permitted until 2026-09-16 19:47:44.334856776 +0000 UTC m=+511.106727165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-9kf4f" (UID: "fa4d18f8-2eb3-4ad0-9448-7b601d74aeab") : secret "plugin-serving-cert" not found
Sep 16 19:47:43.835420 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.835404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:43.850436 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:43.850408    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8scgb\" (UniqueName: \"kubernetes.io/projected/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-kube-api-access-8scgb\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:44.339169 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:44.339129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:44.341476 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:44.341458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa4d18f8-2eb3-4ad0-9448-7b601d74aeab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9kf4f\" (UID: \"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:44.531370 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:44.531328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"
Sep 16 19:47:44.811026 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:44.810996    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f"]
Sep 16 19:47:45.508390 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:45.508341    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f" event={"ID":"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab","Type":"ContainerStarted","Data":"e761bfb32540cf121dcafd46b3962b67d310869a43cfcb08251d58eb8e84044f"}
Sep 16 19:47:49.525646 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:47:49.525618    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f" event={"ID":"fa4d18f8-2eb3-4ad0-9448-7b601d74aeab","Type":"ContainerStarted","Data":"d04fd421a56bad80dceec3965570d6be050890f6985a03f279304ec3666c5945"}
Sep 16 19:48:24.809336 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:24.809228    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9kf4f" podStartSLOduration=37.164095055 podStartE2EDuration="41.809207367s" totalImagesPullingTime="4.645112312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:49.543977562 +0000 UTC m=+516.315847973" watchObservedRunningTime="2026-09-16 19:48:24.809207367 +0000 UTC m=+551.581077780"
Sep 16 19:48:24.809801 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:24.809586    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rmtd5"]
Sep 16 19:48:24.834214 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:24.834185    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rmtd5"]
Sep 16 19:48:24.834352 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:24.834311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:24.837085 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:24.837065    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 19:48:24.902998 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:24.902965    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rmtd5"]
Sep 16 19:48:24.951064 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:24.951034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhcw4\" (UniqueName: \"kubernetes.io/projected/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-kube-api-access-xhcw4\") pod \"limitador-limitador-8588749977-rmtd5\" (UID: \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\") " pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:24.951237 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:24.951105    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-config-file\") pod \"limitador-limitador-8588749977-rmtd5\" (UID: \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\") " pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:25.052399 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.052334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhcw4\" (UniqueName: \"kubernetes.io/projected/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-kube-api-access-xhcw4\") pod \"limitador-limitador-8588749977-rmtd5\" (UID: \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\") " pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:25.052609 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.052453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-config-file\") pod \"limitador-limitador-8588749977-rmtd5\" (UID: \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\") " pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:25.053082 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.053057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-config-file\") pod \"limitador-limitador-8588749977-rmtd5\" (UID: \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\") " pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:25.061814 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.061762    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhcw4\" (UniqueName: \"kubernetes.io/projected/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-kube-api-access-xhcw4\") pod \"limitador-limitador-8588749977-rmtd5\" (UID: \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\") " pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:25.149839 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.149801    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:25.419880 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.419808    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rmtd5"]
Sep 16 19:48:25.423332 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:48:25.423308    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e529c9_8e4c_4ad1_bf8a_55767148b8e7.slice/crio-333598d9ce91a95fc45cb56b43569211b88a99835c88b63a1390bcd79bc4ab21 WatchSource:0}: Error finding container 333598d9ce91a95fc45cb56b43569211b88a99835c88b63a1390bcd79bc4ab21: Status 404 returned error can't find the container with id 333598d9ce91a95fc45cb56b43569211b88a99835c88b63a1390bcd79bc4ab21
Sep 16 19:48:25.660831 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.660789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5" event={"ID":"81e529c9-8e4c-4ad1-bf8a-55767148b8e7","Type":"ContainerStarted","Data":"333598d9ce91a95fc45cb56b43569211b88a99835c88b63a1390bcd79bc4ab21"}
Sep 16 19:48:25.821819 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.821790    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-mjqp2"]
Sep 16 19:48:25.825978 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.825958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mjqp2"
Sep 16 19:48:25.828650 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.828628    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zbh5x\""
Sep 16 19:48:25.831524 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.831499    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mjqp2"]
Sep 16 19:48:25.959128 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:25.959093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltx5\" (UniqueName: \"kubernetes.io/projected/2ee65d12-e040-44ea-925c-1113d598bb88-kube-api-access-9ltx5\") pod \"authorino-59c974bd48-mjqp2\" (UID: \"2ee65d12-e040-44ea-925c-1113d598bb88\") " pod="kuadrant-system/authorino-59c974bd48-mjqp2"
Sep 16 19:48:26.060351 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:26.060315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltx5\" (UniqueName: \"kubernetes.io/projected/2ee65d12-e040-44ea-925c-1113d598bb88-kube-api-access-9ltx5\") pod \"authorino-59c974bd48-mjqp2\" (UID: \"2ee65d12-e040-44ea-925c-1113d598bb88\") " pod="kuadrant-system/authorino-59c974bd48-mjqp2"
Sep 16 19:48:26.069212 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:26.069180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltx5\" (UniqueName: \"kubernetes.io/projected/2ee65d12-e040-44ea-925c-1113d598bb88-kube-api-access-9ltx5\") pod \"authorino-59c974bd48-mjqp2\" (UID: \"2ee65d12-e040-44ea-925c-1113d598bb88\") " pod="kuadrant-system/authorino-59c974bd48-mjqp2"
Sep 16 19:48:26.135521 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:26.135442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mjqp2"
Sep 16 19:48:26.402775 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:26.402693    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mjqp2"]
Sep 16 19:48:26.405611 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:48:26.405582    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ee65d12_e040_44ea_925c_1113d598bb88.slice/crio-5f9241f89b78b12cb42d0aee45d00e06cce97e55aa0289c2db847a865560c4e7 WatchSource:0}: Error finding container 5f9241f89b78b12cb42d0aee45d00e06cce97e55aa0289c2db847a865560c4e7: Status 404 returned error can't find the container with id 5f9241f89b78b12cb42d0aee45d00e06cce97e55aa0289c2db847a865560c4e7
Sep 16 19:48:26.666730 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:26.666644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mjqp2" event={"ID":"2ee65d12-e040-44ea-925c-1113d598bb88","Type":"ContainerStarted","Data":"5f9241f89b78b12cb42d0aee45d00e06cce97e55aa0289c2db847a865560c4e7"}
Sep 16 19:48:28.679398 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:28.679346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5" event={"ID":"81e529c9-8e4c-4ad1-bf8a-55767148b8e7","Type":"ContainerStarted","Data":"23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461"}
Sep 16 19:48:28.679830 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:28.679439    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:29.684528 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:29.684485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mjqp2" event={"ID":"2ee65d12-e040-44ea-925c-1113d598bb88","Type":"ContainerStarted","Data":"ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe"}
Sep 16 19:48:29.718632 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:29.718583    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5" podStartSLOduration=3.166544046 podStartE2EDuration="5.718568482s" totalImagesPullingTime="2.552024436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:28.711786522 +0000 UTC m=+555.483656944" watchObservedRunningTime="2026-09-16 19:48:29.718568482 +0000 UTC m=+556.490438917"
Sep 16 19:48:29.718780 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:29.718664    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-mjqp2" podStartSLOduration=2.194046431 podStartE2EDuration="4.718657912s" totalImagesPullingTime="2.524611481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:29.715691101 +0000 UTC m=+556.487561512" watchObservedRunningTime="2026-09-16 19:48:29.718657912 +0000 UTC m=+556.490528322"
Sep 16 19:48:38.460130 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:38.460094    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rmtd5"]
Sep 16 19:48:38.460585 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:38.460297    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5" podUID="81e529c9-8e4c-4ad1-bf8a-55767148b8e7" containerName="limitador" containerID="cri-o://23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461" gracePeriod=30
Sep 16 19:48:38.461058 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:38.461037    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:38.992782 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:38.992750    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:39.058620 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.058531    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhcw4\" (UniqueName: \"kubernetes.io/projected/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-kube-api-access-xhcw4\") pod \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\" (UID: \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\") "
Sep 16 19:48:39.058620 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.058611    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-config-file\") pod \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\" (UID: \"81e529c9-8e4c-4ad1-bf8a-55767148b8e7\") "
Sep 16 19:48:39.058953 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.058926    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-config-file" (OuterVolumeSpecName: "config-file") pod "81e529c9-8e4c-4ad1-bf8a-55767148b8e7" (UID: "81e529c9-8e4c-4ad1-bf8a-55767148b8e7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:48:39.060589 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.060562    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-kube-api-access-xhcw4" (OuterVolumeSpecName: "kube-api-access-xhcw4") pod "81e529c9-8e4c-4ad1-bf8a-55767148b8e7" (UID: "81e529c9-8e4c-4ad1-bf8a-55767148b8e7"). InnerVolumeSpecName "kube-api-access-xhcw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:39.159883 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.159846    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-config-file\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:48:39.159883 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.159877    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhcw4\" (UniqueName: \"kubernetes.io/projected/81e529c9-8e4c-4ad1-bf8a-55767148b8e7-kube-api-access-xhcw4\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:48:39.723431 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.723399    2561 generic.go:356] "Generic (PLEG): container finished" podID="81e529c9-8e4c-4ad1-bf8a-55767148b8e7" containerID="23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461" exitCode=0
Sep 16 19:48:39.723842 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.723437    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5"
Sep 16 19:48:39.723842 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.723457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5" event={"ID":"81e529c9-8e4c-4ad1-bf8a-55767148b8e7","Type":"ContainerDied","Data":"23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461"}
Sep 16 19:48:39.723842 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.723505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rmtd5" event={"ID":"81e529c9-8e4c-4ad1-bf8a-55767148b8e7","Type":"ContainerDied","Data":"333598d9ce91a95fc45cb56b43569211b88a99835c88b63a1390bcd79bc4ab21"}
Sep 16 19:48:39.723842 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.723529    2561 scope.go:117] "RemoveContainer" containerID="23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461"
Sep 16 19:48:39.731657 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.731636    2561 scope.go:117] "RemoveContainer" containerID="23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461"
Sep 16 19:48:39.731934 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:48:39.731913    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461\": container with ID starting with 23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461 not found: ID does not exist" containerID="23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461"
Sep 16 19:48:39.731995 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.731942    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461"} err="failed to get container status \"23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461\": rpc error: code = NotFound desc = could not find container \"23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461\": container with ID starting with 23cf222d7df7ff8acc638207ed9daf36d61f9325f06d8f02aeefb7b7fa271461 not found: ID does not exist"
Sep 16 19:48:39.743173 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.743148    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rmtd5"]
Sep 16 19:48:39.747344 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:39.747320    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rmtd5"]
Sep 16 19:48:41.682090 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:41.682053    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e529c9-8e4c-4ad1-bf8a-55767148b8e7" path="/var/lib/kubelet/pods/81e529c9-8e4c-4ad1-bf8a-55767148b8e7/volumes"
Sep 16 19:48:48.453054 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.453021    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mjqp2"]
Sep 16 19:48:48.453443 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.453194    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-mjqp2" podUID="2ee65d12-e040-44ea-925c-1113d598bb88" containerName="authorino" containerID="cri-o://ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe" gracePeriod=30
Sep 16 19:48:48.692349 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.692321    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mjqp2"
Sep 16 19:48:48.731110 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.731032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ltx5\" (UniqueName: \"kubernetes.io/projected/2ee65d12-e040-44ea-925c-1113d598bb88-kube-api-access-9ltx5\") pod \"2ee65d12-e040-44ea-925c-1113d598bb88\" (UID: \"2ee65d12-e040-44ea-925c-1113d598bb88\") "
Sep 16 19:48:48.733100 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.733067    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee65d12-e040-44ea-925c-1113d598bb88-kube-api-access-9ltx5" (OuterVolumeSpecName: "kube-api-access-9ltx5") pod "2ee65d12-e040-44ea-925c-1113d598bb88" (UID: "2ee65d12-e040-44ea-925c-1113d598bb88"). InnerVolumeSpecName "kube-api-access-9ltx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:48.760116 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.760086    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ee65d12-e040-44ea-925c-1113d598bb88" containerID="ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe" exitCode=0
Sep 16 19:48:48.760244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.760123    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mjqp2"
Sep 16 19:48:48.760244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.760172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mjqp2" event={"ID":"2ee65d12-e040-44ea-925c-1113d598bb88","Type":"ContainerDied","Data":"ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe"}
Sep 16 19:48:48.760244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.760215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mjqp2" event={"ID":"2ee65d12-e040-44ea-925c-1113d598bb88","Type":"ContainerDied","Data":"5f9241f89b78b12cb42d0aee45d00e06cce97e55aa0289c2db847a865560c4e7"}
Sep 16 19:48:48.760244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.760238    2561 scope.go:117] "RemoveContainer" containerID="ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe"
Sep 16 19:48:48.768739 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.768720    2561 scope.go:117] "RemoveContainer" containerID="ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe"
Sep 16 19:48:48.768992 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:48:48.768974    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe\": container with ID starting with ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe not found: ID does not exist" containerID="ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe"
Sep 16 19:48:48.769051 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.769000    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe"} err="failed to get container status \"ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe\": rpc error: code = NotFound desc = could not find container \"ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe\": container with ID starting with ca519ed86439e1a3b70aaeaa66e969e687f54528847e0afa6472c880f1751efe not found: ID does not exist"
Sep 16 19:48:48.796493 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.796463    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mjqp2"]
Sep 16 19:48:48.804104 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.804082    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mjqp2"]
Sep 16 19:48:48.832394 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:48.832355    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ltx5\" (UniqueName: \"kubernetes.io/projected/2ee65d12-e040-44ea-925c-1113d598bb88-kube-api-access-9ltx5\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:48:49.682358 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:48:49.682326    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee65d12-e040-44ea-925c-1113d598bb88" path="/var/lib/kubelet/pods/2ee65d12-e040-44ea-925c-1113d598bb88/volumes"
Sep 16 19:49:13.617935 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:13.617897    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:49:13.619878 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:13.619857    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:49:24.582295 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.582258    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-7f6bl"]
Sep 16 19:49:24.582766 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.582622    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee65d12-e040-44ea-925c-1113d598bb88" containerName="authorino"
Sep 16 19:49:24.582766 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.582640    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee65d12-e040-44ea-925c-1113d598bb88" containerName="authorino"
Sep 16 19:49:24.582766 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.582659    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e529c9-8e4c-4ad1-bf8a-55767148b8e7" containerName="limitador"
Sep 16 19:49:24.582766 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.582667    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e529c9-8e4c-4ad1-bf8a-55767148b8e7" containerName="limitador"
Sep 16 19:49:24.582766 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.582737    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee65d12-e040-44ea-925c-1113d598bb88" containerName="authorino"
Sep 16 19:49:24.582766 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.582749    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e529c9-8e4c-4ad1-bf8a-55767148b8e7" containerName="limitador"
Sep 16 19:49:24.586600 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.586578    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:24.591006 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.590986    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 19:49:24.591126 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.591063    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 19:49:24.592096 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.592077    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-55z4r\""
Sep 16 19:49:24.592198 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.592102    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 19:49:24.599590 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.599566    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-7f6bl"]
Sep 16 19:49:24.604517 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.604497    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-668c6bff64-76km2"]
Sep 16 19:49:24.607277 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.607262    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:24.609938 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.609909    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 19:49:24.610021 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.609966    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-kt7zl\""
Sep 16 19:49:24.618278 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.618254    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-668c6bff64-76km2"]
Sep 16 19:49:24.700302 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.700263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zmt\" (UniqueName: \"kubernetes.io/projected/f636d596-1ffc-492c-96dd-2b2203f599c9-kube-api-access-z4zmt\") pod \"kserve-controller-manager-7f95f9f864-7f6bl\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") " pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:24.700506 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.700321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b034f221-c36c-4dda-90fb-5bb402d985ae-cert\") pod \"llmisvc-controller-manager-668c6bff64-76km2\" (UID: \"b034f221-c36c-4dda-90fb-5bb402d985ae\") " pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:24.700506 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.700344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwb8d\" (UniqueName: \"kubernetes.io/projected/b034f221-c36c-4dda-90fb-5bb402d985ae-kube-api-access-lwb8d\") pod \"llmisvc-controller-manager-668c6bff64-76km2\" (UID: \"b034f221-c36c-4dda-90fb-5bb402d985ae\") " pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:24.700506 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.700366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert\") pod \"kserve-controller-manager-7f95f9f864-7f6bl\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") " pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:24.801656 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.801619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert\") pod \"kserve-controller-manager-7f95f9f864-7f6bl\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") " pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:24.801656 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.801664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zmt\" (UniqueName: \"kubernetes.io/projected/f636d596-1ffc-492c-96dd-2b2203f599c9-kube-api-access-z4zmt\") pod \"kserve-controller-manager-7f95f9f864-7f6bl\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") " pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:24.801886 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.801728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b034f221-c36c-4dda-90fb-5bb402d985ae-cert\") pod \"llmisvc-controller-manager-668c6bff64-76km2\" (UID: \"b034f221-c36c-4dda-90fb-5bb402d985ae\") " pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:24.801886 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.801745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwb8d\" (UniqueName: \"kubernetes.io/projected/b034f221-c36c-4dda-90fb-5bb402d985ae-kube-api-access-lwb8d\") pod \"llmisvc-controller-manager-668c6bff64-76km2\" (UID: \"b034f221-c36c-4dda-90fb-5bb402d985ae\") " pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:24.801886 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:49:24.801766    2561 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 19:49:24.801886 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:49:24.801847    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert podName:f636d596-1ffc-492c-96dd-2b2203f599c9 nodeName:}" failed. No retries permitted until 2026-09-16 19:49:25.301827316 +0000 UTC m=+612.073697715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert") pod "kserve-controller-manager-7f95f9f864-7f6bl" (UID: "f636d596-1ffc-492c-96dd-2b2203f599c9") : secret "kserve-webhook-server-cert" not found
Sep 16 19:49:24.801886 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:49:24.801866    2561 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 19:49:24.802087 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:49:24.801939    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b034f221-c36c-4dda-90fb-5bb402d985ae-cert podName:b034f221-c36c-4dda-90fb-5bb402d985ae nodeName:}" failed. No retries permitted until 2026-09-16 19:49:25.301920306 +0000 UTC m=+612.073790709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b034f221-c36c-4dda-90fb-5bb402d985ae-cert") pod "llmisvc-controller-manager-668c6bff64-76km2" (UID: "b034f221-c36c-4dda-90fb-5bb402d985ae") : secret "llmisvc-webhook-server-cert" not found
Sep 16 19:49:24.810807 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.810779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zmt\" (UniqueName: \"kubernetes.io/projected/f636d596-1ffc-492c-96dd-2b2203f599c9-kube-api-access-z4zmt\") pod \"kserve-controller-manager-7f95f9f864-7f6bl\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") " pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:24.813473 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:24.813445    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwb8d\" (UniqueName: \"kubernetes.io/projected/b034f221-c36c-4dda-90fb-5bb402d985ae-kube-api-access-lwb8d\") pod \"llmisvc-controller-manager-668c6bff64-76km2\" (UID: \"b034f221-c36c-4dda-90fb-5bb402d985ae\") " pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:25.306037 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.305999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b034f221-c36c-4dda-90fb-5bb402d985ae-cert\") pod \"llmisvc-controller-manager-668c6bff64-76km2\" (UID: \"b034f221-c36c-4dda-90fb-5bb402d985ae\") " pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:25.306216 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.306045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert\") pod \"kserve-controller-manager-7f95f9f864-7f6bl\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") " pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:25.308435 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.308404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert\") pod \"kserve-controller-manager-7f95f9f864-7f6bl\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") " pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:25.308546 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.308445    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b034f221-c36c-4dda-90fb-5bb402d985ae-cert\") pod \"llmisvc-controller-manager-668c6bff64-76km2\" (UID: \"b034f221-c36c-4dda-90fb-5bb402d985ae\") " pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:25.496830 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.496791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:25.516926 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.516900    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:25.766850 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.766827    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-7f6bl"]
Sep 16 19:49:25.769153 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:49:25.769129    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf636d596_1ffc_492c_96dd_2b2203f599c9.slice/crio-f63e6fba859b089f5154ded62b5d34f9be8fd1c9fc9298bc60353a23b85a1409 WatchSource:0}: Error finding container f63e6fba859b089f5154ded62b5d34f9be8fd1c9fc9298bc60353a23b85a1409: Status 404 returned error can't find the container with id f63e6fba859b089f5154ded62b5d34f9be8fd1c9fc9298bc60353a23b85a1409
Sep 16 19:49:25.799898 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.799870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-668c6bff64-76km2"]
Sep 16 19:49:25.802039 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:49:25.802013    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb034f221_c36c_4dda_90fb_5bb402d985ae.slice/crio-e6e5a488ec586bc50309e3eba7e793956661303c820e11cd58f7f668cd814795 WatchSource:0}: Error finding container e6e5a488ec586bc50309e3eba7e793956661303c820e11cd58f7f668cd814795: Status 404 returned error can't find the container with id e6e5a488ec586bc50309e3eba7e793956661303c820e11cd58f7f668cd814795
Sep 16 19:49:25.904915 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.904833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-668c6bff64-76km2" event={"ID":"b034f221-c36c-4dda-90fb-5bb402d985ae","Type":"ContainerStarted","Data":"e6e5a488ec586bc50309e3eba7e793956661303c820e11cd58f7f668cd814795"}
Sep 16 19:49:25.906025 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:25.906002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl" event={"ID":"f636d596-1ffc-492c-96dd-2b2203f599c9","Type":"ContainerStarted","Data":"f63e6fba859b089f5154ded62b5d34f9be8fd1c9fc9298bc60353a23b85a1409"}
Sep 16 19:49:29.930755 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:29.930713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-668c6bff64-76km2" event={"ID":"b034f221-c36c-4dda-90fb-5bb402d985ae","Type":"ContainerStarted","Data":"401d9555b5c5ecd2f810fa8aba42dc90fccc67c5f39beef896df5565673bb259"}
Sep 16 19:49:29.931206 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:29.930875    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:49:29.932297 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:29.932269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl" event={"ID":"f636d596-1ffc-492c-96dd-2b2203f599c9","Type":"ContainerStarted","Data":"e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4"}
Sep 16 19:49:29.932424 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:29.932370    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:49:29.950410 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:29.950349    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-668c6bff64-76km2" podStartSLOduration=2.53659899 podStartE2EDuration="5.95033317s" totalImagesPullingTime="3.41373418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:29.948463282 +0000 UTC m=+616.720333692" watchObservedRunningTime="2026-09-16 19:49:29.95033317 +0000 UTC m=+616.722203580"
Sep 16 19:49:29.968012 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:49:29.967964    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl" podStartSLOduration=2.5180667 podStartE2EDuration="5.967949148s" totalImagesPullingTime="3.449882448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:29.966916115 +0000 UTC m=+616.738786525" watchObservedRunningTime="2026-09-16 19:49:29.967949148 +0000 UTC m=+616.739819559"
Sep 16 19:50:00.938946 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:00.938912    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-668c6bff64-76km2"
Sep 16 19:50:00.942264 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:00.941926    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:50:09.130658 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.130624    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-7f6bl"]
Sep 16 19:50:09.131033 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.130825    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl" podUID="f636d596-1ffc-492c-96dd-2b2203f599c9" containerName="manager" containerID="cri-o://e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4" gracePeriod=10
Sep 16 19:50:09.176131 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.176105    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-c7kdq"]
Sep 16 19:50:09.179152 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.179137    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:09.199952 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.199926    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-c7kdq"]
Sep 16 19:50:09.228730 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.228693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b55zw\" (UniqueName: \"kubernetes.io/projected/eeedf724-46ab-42f8-800e-c88916f4f513-kube-api-access-b55zw\") pod \"kserve-controller-manager-7f95f9f864-c7kdq\" (UID: \"eeedf724-46ab-42f8-800e-c88916f4f513\") " pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:09.228874 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.228786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeedf724-46ab-42f8-800e-c88916f4f513-cert\") pod \"kserve-controller-manager-7f95f9f864-c7kdq\" (UID: \"eeedf724-46ab-42f8-800e-c88916f4f513\") " pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:09.329783 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.329744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeedf724-46ab-42f8-800e-c88916f4f513-cert\") pod \"kserve-controller-manager-7f95f9f864-c7kdq\" (UID: \"eeedf724-46ab-42f8-800e-c88916f4f513\") " pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:09.329783 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.329788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b55zw\" (UniqueName: \"kubernetes.io/projected/eeedf724-46ab-42f8-800e-c88916f4f513-kube-api-access-b55zw\") pod \"kserve-controller-manager-7f95f9f864-c7kdq\" (UID: \"eeedf724-46ab-42f8-800e-c88916f4f513\") " pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:09.331986 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.331956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeedf724-46ab-42f8-800e-c88916f4f513-cert\") pod \"kserve-controller-manager-7f95f9f864-c7kdq\" (UID: \"eeedf724-46ab-42f8-800e-c88916f4f513\") " pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:09.338789 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.338759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b55zw\" (UniqueName: \"kubernetes.io/projected/eeedf724-46ab-42f8-800e-c88916f4f513-kube-api-access-b55zw\") pod \"kserve-controller-manager-7f95f9f864-c7kdq\" (UID: \"eeedf724-46ab-42f8-800e-c88916f4f513\") " pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:09.372581 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.372558    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:50:09.430953 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.430869    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4zmt\" (UniqueName: \"kubernetes.io/projected/f636d596-1ffc-492c-96dd-2b2203f599c9-kube-api-access-z4zmt\") pod \"f636d596-1ffc-492c-96dd-2b2203f599c9\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") "
Sep 16 19:50:09.430953 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.430912    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert\") pod \"f636d596-1ffc-492c-96dd-2b2203f599c9\" (UID: \"f636d596-1ffc-492c-96dd-2b2203f599c9\") "
Sep 16 19:50:09.433053 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.433026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert" (OuterVolumeSpecName: "cert") pod "f636d596-1ffc-492c-96dd-2b2203f599c9" (UID: "f636d596-1ffc-492c-96dd-2b2203f599c9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:50:09.433053 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.433043    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f636d596-1ffc-492c-96dd-2b2203f599c9-kube-api-access-z4zmt" (OuterVolumeSpecName: "kube-api-access-z4zmt") pod "f636d596-1ffc-492c-96dd-2b2203f599c9" (UID: "f636d596-1ffc-492c-96dd-2b2203f599c9"). InnerVolumeSpecName "kube-api-access-z4zmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:09.515680 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.515646    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:09.531818 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.531793    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4zmt\" (UniqueName: \"kubernetes.io/projected/f636d596-1ffc-492c-96dd-2b2203f599c9-kube-api-access-z4zmt\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:50:09.531818 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.531818    2561 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f636d596-1ffc-492c-96dd-2b2203f599c9-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:50:09.787632 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:09.787596    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-c7kdq"]
Sep 16 19:50:09.790639 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:50:09.790599    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeedf724_46ab_42f8_800e_c88916f4f513.slice/crio-ab23e3fe774e078a55c7fd592645a49d46638a471c4d84a237868213b4d769f5 WatchSource:0}: Error finding container ab23e3fe774e078a55c7fd592645a49d46638a471c4d84a237868213b4d769f5: Status 404 returned error can't find the container with id ab23e3fe774e078a55c7fd592645a49d46638a471c4d84a237868213b4d769f5
Sep 16 19:50:10.087422 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.087365    2561 generic.go:356] "Generic (PLEG): container finished" podID="f636d596-1ffc-492c-96dd-2b2203f599c9" containerID="e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4" exitCode=0
Sep 16 19:50:10.087422 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.087423    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl"
Sep 16 19:50:10.087649 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.087449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl" event={"ID":"f636d596-1ffc-492c-96dd-2b2203f599c9","Type":"ContainerDied","Data":"e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4"}
Sep 16 19:50:10.087649 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.087502    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f95f9f864-7f6bl" event={"ID":"f636d596-1ffc-492c-96dd-2b2203f599c9","Type":"ContainerDied","Data":"f63e6fba859b089f5154ded62b5d34f9be8fd1c9fc9298bc60353a23b85a1409"}
Sep 16 19:50:10.087649 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.087531    2561 scope.go:117] "RemoveContainer" containerID="e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4"
Sep 16 19:50:10.088842 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.088809    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq" event={"ID":"eeedf724-46ab-42f8-800e-c88916f4f513","Type":"ContainerStarted","Data":"ab23e3fe774e078a55c7fd592645a49d46638a471c4d84a237868213b4d769f5"}
Sep 16 19:50:10.095461 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.095443    2561 scope.go:117] "RemoveContainer" containerID="e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4"
Sep 16 19:50:10.095746 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:50:10.095723    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4\": container with ID starting with e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4 not found: ID does not exist" containerID="e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4"
Sep 16 19:50:10.095806 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.095755    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4"} err="failed to get container status \"e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4\": rpc error: code = NotFound desc = could not find container \"e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4\": container with ID starting with e86d22f20225425efdcb89434bfb8c9ac8b43c3d9358512abc1119942c673fe4 not found: ID does not exist"
Sep 16 19:50:10.104480 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.104450    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-7f6bl"]
Sep 16 19:50:10.109966 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:10.109941    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7f95f9f864-7f6bl"]
Sep 16 19:50:11.094501 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:11.094464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq" event={"ID":"eeedf724-46ab-42f8-800e-c88916f4f513","Type":"ContainerStarted","Data":"44d61d78290749b520ea263477182b485a76648cd52e91c0328fcdc4382475a6"}
Sep 16 19:50:11.094949 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:11.094530    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:50:11.113087 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:11.113039    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq" podStartSLOduration=1.750628849 podStartE2EDuration="2.113026242s" totalImagesPullingTime="362.397393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:11.111531711 +0000 UTC m=+657.883402122" watchObservedRunningTime="2026-09-16 19:50:11.113026242 +0000 UTC m=+657.884896676"
Sep 16 19:50:11.682930 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:11.682894    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f636d596-1ffc-492c-96dd-2b2203f599c9" path="/var/lib/kubelet/pods/f636d596-1ffc-492c-96dd-2b2203f599c9/volumes"
Sep 16 19:50:42.105559 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:50:42.105520    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f95f9f864-c7kdq"
Sep 16 19:51:00.845945 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.845912    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-p8mb7"]
Sep 16 19:51:00.846405 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.846166    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f636d596-1ffc-492c-96dd-2b2203f599c9" containerName="manager"
Sep 16 19:51:00.846405 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.846176    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f636d596-1ffc-492c-96dd-2b2203f599c9" containerName="manager"
Sep 16 19:51:00.846405 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.846226    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f636d596-1ffc-492c-96dd-2b2203f599c9" containerName="manager"
Sep 16 19:51:00.849178 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.849159    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-p8mb7"
Sep 16 19:51:00.855422 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.851903    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 19:51:00.855422 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.852191    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-h8z9s\""
Sep 16 19:51:00.855422 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.852459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 19:51:00.858606 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:00.858571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-p8mb7"]
Sep 16 19:51:01.018278 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:01.018234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn7q7\" (UniqueName: \"kubernetes.io/projected/7a4c3429-f98b-4971-b3d2-253c66134b42-kube-api-access-rn7q7\") pod \"s3-init-p8mb7\" (UID: \"7a4c3429-f98b-4971-b3d2-253c66134b42\") " pod="kserve/s3-init-p8mb7"
Sep 16 19:51:01.018483 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:01.018319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4c3429-f98b-4971-b3d2-253c66134b42-model-cache\") pod \"s3-init-p8mb7\" (UID: \"7a4c3429-f98b-4971-b3d2-253c66134b42\") " pod="kserve/s3-init-p8mb7"
Sep 16 19:51:01.119601 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:01.119513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn7q7\" (UniqueName: \"kubernetes.io/projected/7a4c3429-f98b-4971-b3d2-253c66134b42-kube-api-access-rn7q7\") pod \"s3-init-p8mb7\" (UID: \"7a4c3429-f98b-4971-b3d2-253c66134b42\") " pod="kserve/s3-init-p8mb7"
Sep 16 19:51:01.119601 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:01.119573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4c3429-f98b-4971-b3d2-253c66134b42-model-cache\") pod \"s3-init-p8mb7\" (UID: \"7a4c3429-f98b-4971-b3d2-253c66134b42\") " pod="kserve/s3-init-p8mb7"
Sep 16 19:51:01.119894 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:01.119879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4c3429-f98b-4971-b3d2-253c66134b42-model-cache\") pod \"s3-init-p8mb7\" (UID: \"7a4c3429-f98b-4971-b3d2-253c66134b42\") " pod="kserve/s3-init-p8mb7"
Sep 16 19:51:01.128140 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:01.128114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn7q7\" (UniqueName: \"kubernetes.io/projected/7a4c3429-f98b-4971-b3d2-253c66134b42-kube-api-access-rn7q7\") pod \"s3-init-p8mb7\" (UID: \"7a4c3429-f98b-4971-b3d2-253c66134b42\") " pod="kserve/s3-init-p8mb7"
Sep 16 19:51:01.165728 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:01.165681    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-p8mb7"
Sep 16 19:51:01.434055 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:01.433971    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-p8mb7"]
Sep 16 19:51:01.436815 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:51:01.436788    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a4c3429_f98b_4971_b3d2_253c66134b42.slice/crio-66d2125b93231a589fcf3f7da75d1e82305612d69d2421ca14e645dcb33b5f00 WatchSource:0}: Error finding container 66d2125b93231a589fcf3f7da75d1e82305612d69d2421ca14e645dcb33b5f00: Status 404 returned error can't find the container with id 66d2125b93231a589fcf3f7da75d1e82305612d69d2421ca14e645dcb33b5f00
Sep 16 19:51:02.296711 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:02.296676    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p8mb7" event={"ID":"7a4c3429-f98b-4971-b3d2-253c66134b42","Type":"ContainerStarted","Data":"66d2125b93231a589fcf3f7da75d1e82305612d69d2421ca14e645dcb33b5f00"}
Sep 16 19:51:05.312812 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:05.312769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p8mb7" event={"ID":"7a4c3429-f98b-4971-b3d2-253c66134b42","Type":"ContainerStarted","Data":"d37cee08a31331fb1afad00d64daa4181053ccd7c0e30875e87c893d3834e679"}
Sep 16 19:51:07.322255 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:07.322218    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a4c3429-f98b-4971-b3d2-253c66134b42" containerID="d37cee08a31331fb1afad00d64daa4181053ccd7c0e30875e87c893d3834e679" exitCode=0
Sep 16 19:51:07.322777 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:07.322298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p8mb7" event={"ID":"7a4c3429-f98b-4971-b3d2-253c66134b42","Type":"ContainerDied","Data":"d37cee08a31331fb1afad00d64daa4181053ccd7c0e30875e87c893d3834e679"}
Sep 16 19:51:12.347391 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:12.347342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p8mb7" event={"ID":"7a4c3429-f98b-4971-b3d2-253c66134b42","Type":"ContainerStarted","Data":"8df6c3d2e89f78131ac408a931ce9c0219b16efdcea8c4e1a4ba2d8cb98f2480"}
Sep 16 19:51:12.370558 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:51:12.370512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-p8mb7" podStartSLOduration=2.6526151000000002 podStartE2EDuration="12.370495273s" totalImagesPullingTime="7.815748344s" totalInitContainerRuntime="1.902131829s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:12.369544392 +0000 UTC m=+719.141414805" watchObservedRunningTime="2026-09-16 19:51:12.370495273 +0000 UTC m=+719.142365684"
Sep 16 19:52:16.603368 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:16.603329    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a4c3429-f98b-4971-b3d2-253c66134b42" containerID="8df6c3d2e89f78131ac408a931ce9c0219b16efdcea8c4e1a4ba2d8cb98f2480" exitCode=0
Sep 16 19:52:16.603801 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:16.603408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p8mb7" event={"ID":"7a4c3429-f98b-4971-b3d2-253c66134b42","Type":"ContainerDied","Data":"8df6c3d2e89f78131ac408a931ce9c0219b16efdcea8c4e1a4ba2d8cb98f2480"}
Sep 16 19:52:17.734251 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:17.734222    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-p8mb7"
Sep 16 19:52:17.894517 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:17.894410    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn7q7\" (UniqueName: \"kubernetes.io/projected/7a4c3429-f98b-4971-b3d2-253c66134b42-kube-api-access-rn7q7\") pod \"7a4c3429-f98b-4971-b3d2-253c66134b42\" (UID: \"7a4c3429-f98b-4971-b3d2-253c66134b42\") "
Sep 16 19:52:17.894517 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:17.894473    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4c3429-f98b-4971-b3d2-253c66134b42-model-cache\") pod \"7a4c3429-f98b-4971-b3d2-253c66134b42\" (UID: \"7a4c3429-f98b-4971-b3d2-253c66134b42\") "
Sep 16 19:52:17.896706 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:17.896677    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4c3429-f98b-4971-b3d2-253c66134b42-kube-api-access-rn7q7" (OuterVolumeSpecName: "kube-api-access-rn7q7") pod "7a4c3429-f98b-4971-b3d2-253c66134b42" (UID: "7a4c3429-f98b-4971-b3d2-253c66134b42"). InnerVolumeSpecName "kube-api-access-rn7q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:52:17.913029 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:17.913004    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4c3429-f98b-4971-b3d2-253c66134b42-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a4c3429-f98b-4971-b3d2-253c66134b42" (UID: "7a4c3429-f98b-4971-b3d2-253c66134b42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:17.995288 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:17.995259    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn7q7\" (UniqueName: \"kubernetes.io/projected/7a4c3429-f98b-4971-b3d2-253c66134b42-kube-api-access-rn7q7\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:52:17.995288 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:17.995282    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4c3429-f98b-4971-b3d2-253c66134b42-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:52:18.614724 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:18.614690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p8mb7" event={"ID":"7a4c3429-f98b-4971-b3d2-253c66134b42","Type":"ContainerDied","Data":"66d2125b93231a589fcf3f7da75d1e82305612d69d2421ca14e645dcb33b5f00"}
Sep 16 19:52:18.614888 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:18.614732    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66d2125b93231a589fcf3f7da75d1e82305612d69d2421ca14e645dcb33b5f00"
Sep 16 19:52:18.614888 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:18.614703    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-p8mb7"
Sep 16 19:52:39.408748 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.408670    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:52:39.409201 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.408948    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a4c3429-f98b-4971-b3d2-253c66134b42" containerName="s3-init"
Sep 16 19:52:39.409201 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.408961    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a4c3429-f98b-4971-b3d2-253c66134b42" containerName="s3-init"
Sep 16 19:52:39.409201 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.408980    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a4c3429-f98b-4971-b3d2-253c66134b42" containerName="download-hf-model"
Sep 16 19:52:39.409201 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.408985    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a4c3429-f98b-4971-b3d2-253c66134b42" containerName="download-hf-model"
Sep 16 19:52:39.409201 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.409032    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a4c3429-f98b-4971-b3d2-253c66134b42" containerName="s3-init"
Sep 16 19:52:39.410820 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.410802    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.414850 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.414831    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 19:52:39.415563 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.415544    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 19:52:39.415638 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.415550    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:52:39.416558 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.416537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-jrdbq\""
Sep 16 19:52:39.416558 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.416538    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 19:52:39.416713 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.416538    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-b9x46\""
Sep 16 19:52:39.416713 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.416586    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 19:52:39.430279 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.430255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:52:39.445236 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.445210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdac4bbe-d790-478d-a9c6-945b0b054dc7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.445400 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.445241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssnzs\" (UniqueName: \"kubernetes.io/projected/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kube-api-access-ssnzs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.445400 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.445264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.445400 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.445370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.445580 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.445434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.445580 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.445481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.445580 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.445514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.445580 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.445554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.546531 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546498    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.546531 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.546802 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.546802 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdac4bbe-d790-478d-a9c6-945b0b054dc7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.546802 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssnzs\" (UniqueName: \"kubernetes.io/projected/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kube-api-access-ssnzs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.546802 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.546802 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.546802 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.547091 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.546949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.547091 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.547070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.547176 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.547150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.547321 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.547301    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.547369 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.547347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdac4bbe-d790-478d-a9c6-945b0b054dc7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.548886 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.548865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.549078 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.549061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.559495 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.559469    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssnzs\" (UniqueName: \"kubernetes.io/projected/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kube-api-access-ssnzs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.721243 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.721152    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:52:39.996336 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:39.996286    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:52:39.998499 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:52:39.998473    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdac4bbe_d790_478d_a9c6_945b0b054dc7.slice/crio-d62ca0362ddda0208017eb20524feacb0d8248f7cdf85055086769028baef629 WatchSource:0}: Error finding container d62ca0362ddda0208017eb20524feacb0d8248f7cdf85055086769028baef629: Status 404 returned error can't find the container with id d62ca0362ddda0208017eb20524feacb0d8248f7cdf85055086769028baef629
Sep 16 19:52:40.705798 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:40.705762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdac4bbe-d790-478d-a9c6-945b0b054dc7","Type":"ContainerStarted","Data":"cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993"}
Sep 16 19:52:40.706168 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:40.705823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdac4bbe-d790-478d-a9c6-945b0b054dc7","Type":"ContainerStarted","Data":"d62ca0362ddda0208017eb20524feacb0d8248f7cdf85055086769028baef629"}
Sep 16 19:52:42.715577 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:42.715537    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerID="cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993" exitCode=0
Sep 16 19:52:42.715937 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:42.715612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdac4bbe-d790-478d-a9c6-945b0b054dc7","Type":"ContainerDied","Data":"cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993"}
Sep 16 19:52:42.717208 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:42.717193    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:52:53.211742 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.210597    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"]
Sep 16 19:52:53.491192 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.491101    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"]
Sep 16 19:52:53.491340 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.491228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:53.494369 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.494343    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 19:52:53.495749 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.495726    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 19:52:53.495950 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.495930    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 19:52:53.496047 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.495995    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-f8bw9\""
Sep 16 19:52:53.496107 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.496050    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-7rb5g\""
Sep 16 19:52:53.567250 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.567205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqswv\" (UniqueName: \"kubernetes.io/projected/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-kube-api-access-kqswv\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8\" (UID: \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:53.567441 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.567270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8\" (UID: \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:53.668598 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.668554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqswv\" (UniqueName: \"kubernetes.io/projected/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-kube-api-access-kqswv\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8\" (UID: \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:53.668770 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.668618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8\" (UID: \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:53.671359 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.671331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8\" (UID: \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:53.677100 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.676886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqswv\" (UniqueName: \"kubernetes.io/projected/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-kube-api-access-kqswv\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8\" (UID: \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:53.804326 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:53.804292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:55.226336 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:55.226300    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"]
Sep 16 19:52:55.228892 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:52:55.228860    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d1a25b_15f0_4d66_95d4_9f0806e61cb0.slice/crio-14aa6ca675eb695774e4811876493c4fa478574a0420c8d7ddb1eb94a2b2b922 WatchSource:0}: Error finding container 14aa6ca675eb695774e4811876493c4fa478574a0420c8d7ddb1eb94a2b2b922: Status 404 returned error can't find the container with id 14aa6ca675eb695774e4811876493c4fa478574a0420c8d7ddb1eb94a2b2b922
Sep 16 19:52:55.791705 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:55.791671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" event={"ID":"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0","Type":"ContainerStarted","Data":"14aa6ca675eb695774e4811876493c4fa478574a0420c8d7ddb1eb94a2b2b922"}
Sep 16 19:52:58.808282 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:58.808240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" event={"ID":"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0","Type":"ContainerStarted","Data":"de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8"}
Sep 16 19:52:58.808742 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:58.808359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:52:58.825199 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:52:58.825149    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" podStartSLOduration=2.727613475 podStartE2EDuration="5.825132147s" totalImagesPullingTime="3.097518672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:58.823824522 +0000 UTC m=+825.595694935" watchObservedRunningTime="2026-09-16 19:52:58.825132147 +0000 UTC m=+825.597002558"
Sep 16 19:53:12.875412 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:12.875356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdac4bbe-d790-478d-a9c6-945b0b054dc7","Type":"ContainerStarted","Data":"b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef"}
Sep 16 19:53:12.900675 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:12.900602    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.655446005 podStartE2EDuration="33.900579929s" totalImagesPullingTime="29.084954522s" totalInitContainerRuntime="2.160179402s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:53:12.897161077 +0000 UTC m=+839.669031485" watchObservedRunningTime="2026-09-16 19:53:12.900579929 +0000 UTC m=+839.672450341"
Sep 16 19:53:13.721597 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:13.721566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:53:19.722161 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:19.722119    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:53:19.723645 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:19.723615    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:53:29.722105 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:29.722062    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:53:29.815536 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:29.815506    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:53:36.621767 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:36.621715    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"]
Sep 16 19:53:39.721857 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:39.721795    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:53:39.815677 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:39.815633    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:49.722108 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:49.722050    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:53:49.815823 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:49.815778    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:51.642024 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:51.641925    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main" containerID="cri-o://de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8" gracePeriod=45
Sep 16 19:53:59.722469 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:59.722425    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:53:59.815630 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:59.815589    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:59.815784 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:53:59.815728    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:54:09.721886 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:09.721789    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:54:09.815459 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:09.815427    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:13.644632 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:13.644597    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:54:13.646236 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:13.646215    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:54:19.721996 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:19.721943    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:54:19.815067 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:19.815028    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:21.894124 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:21.894095    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:54:21.990031 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:21.989945    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqswv\" (UniqueName: \"kubernetes.io/projected/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-kube-api-access-kqswv\") pod \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\" (UID: \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\") "
Sep 16 19:54:21.990031 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:21.990005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-tls-certs\") pod \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\" (UID: \"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0\") "
Sep 16 19:54:21.992257 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:21.992223    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" (UID: "b9d1a25b-15f0-4d66-95d4-9f0806e61cb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:21.992257 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:21.992243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-kube-api-access-kqswv" (OuterVolumeSpecName: "kube-api-access-kqswv") pod "b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" (UID: "b9d1a25b-15f0-4d66-95d4-9f0806e61cb0"). InnerVolumeSpecName "kube-api-access-kqswv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:22.091069 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.091025    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:54:22.091069 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.091070    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqswv\" (UniqueName: \"kubernetes.io/projected/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0-kube-api-access-kqswv\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:54:22.171746 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.171710    2561 generic.go:356] "Generic (PLEG): container finished" podID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerID="de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8" exitCode=0
Sep 16 19:54:22.171746 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.171750    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" event={"ID":"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0","Type":"ContainerDied","Data":"de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8"}
Sep 16 19:54:22.172042 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.171776    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"
Sep 16 19:54:22.172042 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.171799    2561 scope.go:117] "RemoveContainer" containerID="de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8"
Sep 16 19:54:22.172042 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.171789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8" event={"ID":"b9d1a25b-15f0-4d66-95d4-9f0806e61cb0","Type":"ContainerDied","Data":"14aa6ca675eb695774e4811876493c4fa478574a0420c8d7ddb1eb94a2b2b922"}
Sep 16 19:54:22.179755 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.179735    2561 scope.go:117] "RemoveContainer" containerID="de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8"
Sep 16 19:54:22.179997 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:54:22.179977    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8\": container with ID starting with de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8 not found: ID does not exist" containerID="de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8"
Sep 16 19:54:22.180054 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.180006    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8"} err="failed to get container status \"de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8\": rpc error: code = NotFound desc = could not find container \"de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8\": container with ID starting with de55a9305687adfa5eb08a3cd095e7398755e09529abd85bb4fa2786cbd297d8 not found: ID does not exist"
Sep 16 19:54:22.198643 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.198615    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"]
Sep 16 19:54:22.202904 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:22.202882    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-zw6d8"]
Sep 16 19:54:23.683712 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:23.683679    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" path="/var/lib/kubelet/pods/b9d1a25b-15f0-4d66-95d4-9f0806e61cb0/volumes"
Sep 16 19:54:29.721857 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:29.721806    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:54:39.722180 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:39.722128    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:54:49.732221 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:49.732186    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:54:49.739921 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:49.739885    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:54:55.687308 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.687272    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"]
Sep 16 19:54:55.688402 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.688347    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main"
Sep 16 19:54:55.688402 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.688395    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main"
Sep 16 19:54:55.688608 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.688527    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d1a25b-15f0-4d66-95d4-9f0806e61cb0" containerName="main"
Sep 16 19:54:55.692260 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.692239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:55.695032 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.694997    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 19:54:55.696451 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.696433    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 19:54:55.696561 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.696542    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-dn99m\""
Sep 16 19:54:55.696639 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.696624    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-6dsn6\""
Sep 16 19:54:55.696706 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.696691    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 19:54:55.700033 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.700011    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"]
Sep 16 19:54:55.764417 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.764366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60a6825-9602-4ea8-9660-9632e679d348-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz\" (UID: \"b60a6825-9602-4ea8-9660-9632e679d348\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:55.764583 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.764465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgwbp\" (UniqueName: \"kubernetes.io/projected/b60a6825-9602-4ea8-9660-9632e679d348-kube-api-access-rgwbp\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz\" (UID: \"b60a6825-9602-4ea8-9660-9632e679d348\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:55.864766 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.864734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgwbp\" (UniqueName: \"kubernetes.io/projected/b60a6825-9602-4ea8-9660-9632e679d348-kube-api-access-rgwbp\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz\" (UID: \"b60a6825-9602-4ea8-9660-9632e679d348\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:55.864971 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.864784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60a6825-9602-4ea8-9660-9632e679d348-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz\" (UID: \"b60a6825-9602-4ea8-9660-9632e679d348\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:55.867185 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.867159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60a6825-9602-4ea8-9660-9632e679d348-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz\" (UID: \"b60a6825-9602-4ea8-9660-9632e679d348\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:55.872741 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:55.872713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgwbp\" (UniqueName: \"kubernetes.io/projected/b60a6825-9602-4ea8-9660-9632e679d348-kube-api-access-rgwbp\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz\" (UID: \"b60a6825-9602-4ea8-9660-9632e679d348\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:56.003555 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:56.003519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:56.273463 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:56.273434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"]
Sep 16 19:54:56.277066 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:54:56.277039    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60a6825_9602_4ea8_9660_9632e679d348.slice/crio-9ef4404891a8a9db922aca02815919358259ae4c93843095d6682cd04b7cb10c WatchSource:0}: Error finding container 9ef4404891a8a9db922aca02815919358259ae4c93843095d6682cd04b7cb10c: Status 404 returned error can't find the container with id 9ef4404891a8a9db922aca02815919358259ae4c93843095d6682cd04b7cb10c
Sep 16 19:54:56.325550 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:56.325481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" event={"ID":"b60a6825-9602-4ea8-9660-9632e679d348","Type":"ContainerStarted","Data":"9ef4404891a8a9db922aca02815919358259ae4c93843095d6682cd04b7cb10c"}
Sep 16 19:54:57.332425 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:57.332367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" event={"ID":"b60a6825-9602-4ea8-9660-9632e679d348","Type":"ContainerStarted","Data":"f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899"}
Sep 16 19:54:57.332834 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:57.332473    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:54:57.349760 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:54:57.349708    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" podStartSLOduration=2.349692263 podStartE2EDuration="2.349692263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:57.349548772 +0000 UTC m=+944.121419182" watchObservedRunningTime="2026-09-16 19:54:57.349692263 +0000 UTC m=+944.121562673"
Sep 16 19:55:28.338948 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:28.338919    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:55:30.461806 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:30.461771    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:55:35.025965 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:35.025929    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"]
Sep 16 19:55:38.339597 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:38.339512    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:45.484431 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:45.484348    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" containerID="cri-o://b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef" gracePeriod=45
Sep 16 19:55:45.733806 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:45.733768    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": dial tcp 10.132.0.24:8000: connect: connection refused"
Sep 16 19:55:46.225441 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.225417    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:55:46.344910 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.344821    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-dshm\") pod \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") "
Sep 16 19:55:46.344910 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.344865    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdac4bbe-d790-478d-a9c6-945b0b054dc7-cabundle-cert\") pod \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") "
Sep 16 19:55:46.345107 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.344922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tmp-dir\") pod \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") "
Sep 16 19:55:46.345107 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.344944    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kserve-provision-location\") pod \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") "
Sep 16 19:55:46.345107 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.344966    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tls-certs\") pod \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") "
Sep 16 19:55:46.345107 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.345005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-model-cache\") pod \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") "
Sep 16 19:55:46.345107 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.345037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssnzs\" (UniqueName: \"kubernetes.io/projected/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kube-api-access-ssnzs\") pod \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") "
Sep 16 19:55:46.345107 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.345060    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-home\") pod \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\" (UID: \"cdac4bbe-d790-478d-a9c6-945b0b054dc7\") "
Sep 16 19:55:46.345941 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.345613    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdac4bbe-d790-478d-a9c6-945b0b054dc7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cdac4bbe-d790-478d-a9c6-945b0b054dc7" (UID: "cdac4bbe-d790-478d-a9c6-945b0b054dc7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:55:46.345941 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.345752    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdac4bbe-d790-478d-a9c6-945b0b054dc7" (UID: "cdac4bbe-d790-478d-a9c6-945b0b054dc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:46.346571 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.346539    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-home" (OuterVolumeSpecName: "home") pod "cdac4bbe-d790-478d-a9c6-945b0b054dc7" (UID: "cdac4bbe-d790-478d-a9c6-945b0b054dc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:46.347043 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.347024    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:55:46.347159 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.347147    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:55:46.347250 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.347239    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdac4bbe-d790-478d-a9c6-945b0b054dc7-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:55:46.348838 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.348797    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdac4bbe-d790-478d-a9c6-945b0b054dc7" (UID: "cdac4bbe-d790-478d-a9c6-945b0b054dc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:55:46.354298 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.354169    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kube-api-access-ssnzs" (OuterVolumeSpecName: "kube-api-access-ssnzs") pod "cdac4bbe-d790-478d-a9c6-945b0b054dc7" (UID: "cdac4bbe-d790-478d-a9c6-945b0b054dc7"). InnerVolumeSpecName "kube-api-access-ssnzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:55:46.354566 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.354538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-dshm" (OuterVolumeSpecName: "dshm") pod "cdac4bbe-d790-478d-a9c6-945b0b054dc7" (UID: "cdac4bbe-d790-478d-a9c6-945b0b054dc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:46.363711 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.363679    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdac4bbe-d790-478d-a9c6-945b0b054dc7" (UID: "cdac4bbe-d790-478d-a9c6-945b0b054dc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:46.371696 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.371674    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdac4bbe-d790-478d-a9c6-945b0b054dc7" (UID: "cdac4bbe-d790-478d-a9c6-945b0b054dc7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:46.448236 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.448204    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssnzs\" (UniqueName: \"kubernetes.io/projected/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kube-api-access-ssnzs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:55:46.448236 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.448232    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:55:46.448236 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.448242    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:55:46.448480 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.448251    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdac4bbe-d790-478d-a9c6-945b0b054dc7-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:55:46.448480 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.448259    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdac4bbe-d790-478d-a9c6-945b0b054dc7-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:55:46.543558 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.543515    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerID="b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef" exitCode=0
Sep 16 19:55:46.544006 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.543562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdac4bbe-d790-478d-a9c6-945b0b054dc7","Type":"ContainerDied","Data":"b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef"}
Sep 16 19:55:46.544006 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.543586    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:55:46.544006 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.543652    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdac4bbe-d790-478d-a9c6-945b0b054dc7","Type":"ContainerDied","Data":"d62ca0362ddda0208017eb20524feacb0d8248f7cdf85055086769028baef629"}
Sep 16 19:55:46.544006 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.543692    2561 scope.go:117] "RemoveContainer" containerID="b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef"
Sep 16 19:55:46.551902 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.551882    2561 scope.go:117] "RemoveContainer" containerID="cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993"
Sep 16 19:55:46.559156 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.559137    2561 scope.go:117] "RemoveContainer" containerID="b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef"
Sep 16 19:55:46.559428 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:55:46.559409    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef\": container with ID starting with b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef not found: ID does not exist" containerID="b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef"
Sep 16 19:55:46.559480 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.559436    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef"} err="failed to get container status \"b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef\": rpc error: code = NotFound desc = could not find container \"b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef\": container with ID starting with b4fc24d9d6a820f60ec510530d82de0fdb3f3bed2c6ce61bd9f6eb29c779b6ef not found: ID does not exist"
Sep 16 19:55:46.559480 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.559452    2561 scope.go:117] "RemoveContainer" containerID="cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993"
Sep 16 19:55:46.559690 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:55:46.559673    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993\": container with ID starting with cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993 not found: ID does not exist" containerID="cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993"
Sep 16 19:55:46.559742 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.559700    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993"} err="failed to get container status \"cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993\": rpc error: code = NotFound desc = could not find container \"cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993\": container with ID starting with cf785244568f0b1e0883679c57436e580bb18fa3e7be8ae1ebdf491c4c828993 not found: ID does not exist"
Sep 16 19:55:46.564903 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.564878    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:55:46.568471 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:46.568450    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:55:47.682603 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:47.682570    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" path="/var/lib/kubelet/pods/cdac4bbe-d790-478d-a9c6-945b0b054dc7/volumes"
Sep 16 19:55:48.339164 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:48.339130    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:50.045239 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:50.045195    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main" containerID="cri-o://f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899" gracePeriod=45
Sep 16 19:55:58.339285 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:58.339247    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:58.339695 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:55:58.339352    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:56:08.338670 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:08.338627    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:18.338609 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:18.338568    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:20.287712 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.287687    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:56:20.404662 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.404577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgwbp\" (UniqueName: \"kubernetes.io/projected/b60a6825-9602-4ea8-9660-9632e679d348-kube-api-access-rgwbp\") pod \"b60a6825-9602-4ea8-9660-9632e679d348\" (UID: \"b60a6825-9602-4ea8-9660-9632e679d348\") "
Sep 16 19:56:20.404662 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.404653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60a6825-9602-4ea8-9660-9632e679d348-tls-certs\") pod \"b60a6825-9602-4ea8-9660-9632e679d348\" (UID: \"b60a6825-9602-4ea8-9660-9632e679d348\") "
Sep 16 19:56:20.406817 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.406782    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60a6825-9602-4ea8-9660-9632e679d348-kube-api-access-rgwbp" (OuterVolumeSpecName: "kube-api-access-rgwbp") pod "b60a6825-9602-4ea8-9660-9632e679d348" (UID: "b60a6825-9602-4ea8-9660-9632e679d348"). InnerVolumeSpecName "kube-api-access-rgwbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:20.406817 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.406781    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60a6825-9602-4ea8-9660-9632e679d348-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b60a6825-9602-4ea8-9660-9632e679d348" (UID: "b60a6825-9602-4ea8-9660-9632e679d348"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:20.505193 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.505164    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgwbp\" (UniqueName: \"kubernetes.io/projected/b60a6825-9602-4ea8-9660-9632e679d348-kube-api-access-rgwbp\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:56:20.505193 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.505190    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60a6825-9602-4ea8-9660-9632e679d348-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:56:20.689205 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.689117    2561 generic.go:356] "Generic (PLEG): container finished" podID="b60a6825-9602-4ea8-9660-9632e679d348" containerID="f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899" exitCode=0
Sep 16 19:56:20.689205 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.689162    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"
Sep 16 19:56:20.689205 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.689176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" event={"ID":"b60a6825-9602-4ea8-9660-9632e679d348","Type":"ContainerDied","Data":"f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899"}
Sep 16 19:56:20.689444 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.689223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz" event={"ID":"b60a6825-9602-4ea8-9660-9632e679d348","Type":"ContainerDied","Data":"9ef4404891a8a9db922aca02815919358259ae4c93843095d6682cd04b7cb10c"}
Sep 16 19:56:20.689444 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.689247    2561 scope.go:117] "RemoveContainer" containerID="f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899"
Sep 16 19:56:20.697808 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.697789    2561 scope.go:117] "RemoveContainer" containerID="f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899"
Sep 16 19:56:20.698075 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:56:20.698053    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899\": container with ID starting with f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899 not found: ID does not exist" containerID="f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899"
Sep 16 19:56:20.698119 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.698084    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899"} err="failed to get container status \"f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899\": rpc error: code = NotFound desc = could not find container \"f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899\": container with ID starting with f83b20f6c2ad51a020159b80629c14ef1a03dcfa3c18a73abdb13c3f36154899 not found: ID does not exist"
Sep 16 19:56:20.709656 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.709633    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"]
Sep 16 19:56:20.715745 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:20.715724    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-hj5gz"]
Sep 16 19:56:21.682523 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:21.682489    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60a6825-9602-4ea8-9660-9632e679d348" path="/var/lib/kubelet/pods/b60a6825-9602-4ea8-9660-9632e679d348/volumes"
Sep 16 19:56:33.376708 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.376675    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"]
Sep 16 19:56:33.377074 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.376950    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="storage-initializer"
Sep 16 19:56:33.377074 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.376962    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="storage-initializer"
Sep 16 19:56:33.377074 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.376977    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main"
Sep 16 19:56:33.377074 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.376985    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main"
Sep 16 19:56:33.377074 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.376996    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main"
Sep 16 19:56:33.377074 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.377002    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main"
Sep 16 19:56:33.377074 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.377047    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdac4bbe-d790-478d-a9c6-945b0b054dc7" containerName="main"
Sep 16 19:56:33.377074 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.377054    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60a6825-9602-4ea8-9660-9632e679d348" containerName="main"
Sep 16 19:56:33.381328 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.381304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.385560 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.385534    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-cc2hp\""
Sep 16 19:56:33.385703 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.385536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 19:56:33.385703 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.385536    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 19:56:33.385703 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.385537    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 19:56:33.389056 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.388831    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"]
Sep 16 19:56:33.400731 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.400704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67d2b97-0ad6-4c26-8390-f05219a70cff-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.400859 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.400760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8f6n\" (UniqueName: \"kubernetes.io/projected/b67d2b97-0ad6-4c26-8390-f05219a70cff-kube-api-access-b8f6n\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.400859 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.400813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.400859 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.400854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.401007 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.400897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.401007 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.400937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.501474 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.501441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.501676 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.501489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.501676 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.501533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67d2b97-0ad6-4c26-8390-f05219a70cff-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.501676 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.501565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8f6n\" (UniqueName: \"kubernetes.io/projected/b67d2b97-0ad6-4c26-8390-f05219a70cff-kube-api-access-b8f6n\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.501676 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.501594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.501877 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.501709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.501955 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.501901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.501955 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.501940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.502111 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.502090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.503906 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.503880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.504112 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.504093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67d2b97-0ad6-4c26-8390-f05219a70cff-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.509486 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.509463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8f6n\" (UniqueName: \"kubernetes.io/projected/b67d2b97-0ad6-4c26-8390-f05219a70cff-kube-api-access-b8f6n\") pod \"scheduler-inline-config-test-kserve-8655bc757-c7zdr\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.692912 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.692829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:33.964073 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:33.963976    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"]
Sep 16 19:56:33.966159 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:56:33.966131    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67d2b97_0ad6_4c26_8390_f05219a70cff.slice/crio-1a6f542ab43fb1bbec33c4e43e71c6f164a1b01c09ec29c0f55c951697c63877 WatchSource:0}: Error finding container 1a6f542ab43fb1bbec33c4e43e71c6f164a1b01c09ec29c0f55c951697c63877: Status 404 returned error can't find the container with id 1a6f542ab43fb1bbec33c4e43e71c6f164a1b01c09ec29c0f55c951697c63877
Sep 16 19:56:34.755936 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:34.755884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" event={"ID":"b67d2b97-0ad6-4c26-8390-f05219a70cff","Type":"ContainerStarted","Data":"1a6f542ab43fb1bbec33c4e43e71c6f164a1b01c09ec29c0f55c951697c63877"}
Sep 16 19:56:35.761505 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:35.761415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" event={"ID":"b67d2b97-0ad6-4c26-8390-f05219a70cff","Type":"ContainerStarted","Data":"d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc"}
Sep 16 19:56:35.782349 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:35.782293    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" podStartSLOduration=1.245328575 podStartE2EDuration="2.782276767s" totalImagesPullingTime="1.536948192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:56:35.780750267 +0000 UTC m=+1042.552620678" watchObservedRunningTime="2026-09-16 19:56:35.782276767 +0000 UTC m=+1042.554147181"
Sep 16 19:56:36.693115 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:36.693079    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:43.693772 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:43.693740    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:43.706090 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:43.706053    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:56:43.802673 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:56:43.802638    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:57:24.750244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:24.750211    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"]
Sep 16 19:57:24.758915 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:57:24.758873    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:57:24.758915 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:24.758915 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:24.758915 ip-10-0-136-120 kubenswrapper[2561]:  > containerID="d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc" cmd=["/bin/sleep","15"]
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:24.758931    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" message="" err=<
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]:  >
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:57:24.758949    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" podUID="b67d2b97-0ad6-4c26-8390-f05219a70cff" containerName="main" containerID="cri-o://d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc"
Sep 16 19:57:24.759188 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:24.758992    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" podUID="b67d2b97-0ad6-4c26-8390-f05219a70cff" containerName="main" containerID="cri-o://d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc" gracePeriod=60
Sep 16 19:57:24.791888 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:24.791850    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" podUID="b67d2b97-0ad6-4c26-8390-f05219a70cff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.27:8000/health\": dial tcp 10.132.0.27:8000: connect: connection refused"
Sep 16 19:57:25.100241 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.100214    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:57:25.206958 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.206923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-tmp-dir\") pod \"b67d2b97-0ad6-4c26-8390-f05219a70cff\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") "
Sep 16 19:57:25.206958 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.206964    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-home\") pod \"b67d2b97-0ad6-4c26-8390-f05219a70cff\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") "
Sep 16 19:57:25.207189 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.206997    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-dshm\") pod \"b67d2b97-0ad6-4c26-8390-f05219a70cff\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") "
Sep 16 19:57:25.207189 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.207056    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8f6n\" (UniqueName: \"kubernetes.io/projected/b67d2b97-0ad6-4c26-8390-f05219a70cff-kube-api-access-b8f6n\") pod \"b67d2b97-0ad6-4c26-8390-f05219a70cff\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") "
Sep 16 19:57:25.207189 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.207087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-model-cache\") pod \"b67d2b97-0ad6-4c26-8390-f05219a70cff\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") "
Sep 16 19:57:25.207189 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.207120    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67d2b97-0ad6-4c26-8390-f05219a70cff-tls-certs\") pod \"b67d2b97-0ad6-4c26-8390-f05219a70cff\" (UID: \"b67d2b97-0ad6-4c26-8390-f05219a70cff\") "
Sep 16 19:57:25.207439 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.207232    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-home" (OuterVolumeSpecName: "home") pod "b67d2b97-0ad6-4c26-8390-f05219a70cff" (UID: "b67d2b97-0ad6-4c26-8390-f05219a70cff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:25.207439 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.207243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b67d2b97-0ad6-4c26-8390-f05219a70cff" (UID: "b67d2b97-0ad6-4c26-8390-f05219a70cff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:25.207439 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.207352    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:57:25.207439 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.207367    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:57:25.207590 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.207454    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-model-cache" (OuterVolumeSpecName: "model-cache") pod "b67d2b97-0ad6-4c26-8390-f05219a70cff" (UID: "b67d2b97-0ad6-4c26-8390-f05219a70cff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:25.209320 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.209301    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-dshm" (OuterVolumeSpecName: "dshm") pod "b67d2b97-0ad6-4c26-8390-f05219a70cff" (UID: "b67d2b97-0ad6-4c26-8390-f05219a70cff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:25.209420 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.209307    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67d2b97-0ad6-4c26-8390-f05219a70cff-kube-api-access-b8f6n" (OuterVolumeSpecName: "kube-api-access-b8f6n") pod "b67d2b97-0ad6-4c26-8390-f05219a70cff" (UID: "b67d2b97-0ad6-4c26-8390-f05219a70cff"). InnerVolumeSpecName "kube-api-access-b8f6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:25.209420 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.209395    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67d2b97-0ad6-4c26-8390-f05219a70cff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b67d2b97-0ad6-4c26-8390-f05219a70cff" (UID: "b67d2b97-0ad6-4c26-8390-f05219a70cff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:25.307982 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.307889    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:57:25.307982 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.307924    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8f6n\" (UniqueName: \"kubernetes.io/projected/b67d2b97-0ad6-4c26-8390-f05219a70cff-kube-api-access-b8f6n\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:57:25.307982 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.307936    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b67d2b97-0ad6-4c26-8390-f05219a70cff-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:57:25.307982 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.307945    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b67d2b97-0ad6-4c26-8390-f05219a70cff-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 19:57:25.966031 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.965939    2561 generic.go:356] "Generic (PLEG): container finished" podID="b67d2b97-0ad6-4c26-8390-f05219a70cff" containerID="d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc" exitCode=0
Sep 16 19:57:25.966031 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.965991    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"
Sep 16 19:57:25.966031 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.966013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" event={"ID":"b67d2b97-0ad6-4c26-8390-f05219a70cff","Type":"ContainerDied","Data":"d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc"}
Sep 16 19:57:25.966615 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.966061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr" event={"ID":"b67d2b97-0ad6-4c26-8390-f05219a70cff","Type":"ContainerDied","Data":"1a6f542ab43fb1bbec33c4e43e71c6f164a1b01c09ec29c0f55c951697c63877"}
Sep 16 19:57:25.966615 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.966084    2561 scope.go:117] "RemoveContainer" containerID="d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc"
Sep 16 19:57:25.974588 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.974568    2561 scope.go:117] "RemoveContainer" containerID="d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc"
Sep 16 19:57:25.974865 ip-10-0-136-120 kubenswrapper[2561]: E0916 19:57:25.974845    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc\": container with ID starting with d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc not found: ID does not exist" containerID="d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc"
Sep 16 19:57:25.974933 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.974876    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc"} err="failed to get container status \"d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc\": rpc error: code = NotFound desc = could not find container \"d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc\": container with ID starting with d3466b24a326e123ace2a05a159907e0c43ec7425f0851215616266ec15409bc not found: ID does not exist"
Sep 16 19:57:25.981999 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.981971    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"]
Sep 16 19:57:25.987085 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:25.987065    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-c7zdr"]
Sep 16 19:57:27.682764 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:27.682731    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b67d2b97-0ad6-4c26-8390-f05219a70cff" path="/var/lib/kubelet/pods/b67d2b97-0ad6-4c26-8390-f05219a70cff/volumes"
Sep 16 19:57:40.899484 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.899449    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"]
Sep 16 19:57:40.899857 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.899734    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b67d2b97-0ad6-4c26-8390-f05219a70cff" containerName="main"
Sep 16 19:57:40.899857 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.899748    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b67d2b97-0ad6-4c26-8390-f05219a70cff" containerName="main"
Sep 16 19:57:40.899857 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.899810    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b67d2b97-0ad6-4c26-8390-f05219a70cff" containerName="main"
Sep 16 19:57:40.904112 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.904094    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:40.906890 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.906860    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 19:57:40.907029 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.907007    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 19:57:40.907094 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.907007    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:57:40.907147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.907012    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 19:57:40.907147 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.907101    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-47n7j\""
Sep 16 19:57:40.907354 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.907337    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 19:57:40.915243 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:40.915220    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"]
Sep 16 19:57:41.020763 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.020726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-tmp-dir\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.020763 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.020764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-model-cache\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.020997 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.020787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5a0328-43a1-45bb-ba27-23dc334f1efd-tls-certs\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.020997 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.020846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-kserve-provision-location\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.020997 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.020901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmb8\" (UniqueName: \"kubernetes.io/projected/de5a0328-43a1-45bb-ba27-23dc334f1efd-kube-api-access-hsmb8\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.020997 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.020920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5a0328-43a1-45bb-ba27-23dc334f1efd-cabundle-cert\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.020997 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.020937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-home\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.020997 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.020980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-dshm\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.121914 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.121867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5a0328-43a1-45bb-ba27-23dc334f1efd-cabundle-cert\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.121914 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.121918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-home\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122175 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.121946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-dshm\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122175 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.121993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-tmp-dir\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122175 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-model-cache\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122175 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5a0328-43a1-45bb-ba27-23dc334f1efd-tls-certs\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122359 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-kserve-provision-location\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122359 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmb8\" (UniqueName: \"kubernetes.io/projected/de5a0328-43a1-45bb-ba27-23dc334f1efd-kube-api-access-hsmb8\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122492 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-home\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122492 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-model-cache\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122662 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-tmp-dir\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122803 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-kserve-provision-location\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.122911 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.122891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5a0328-43a1-45bb-ba27-23dc334f1efd-cabundle-cert\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.124496 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.124479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-dshm\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.124600 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.124583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5a0328-43a1-45bb-ba27-23dc334f1efd-tls-certs\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.131161 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.131132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmb8\" (UniqueName: \"kubernetes.io/projected/de5a0328-43a1-45bb-ba27-23dc334f1efd-kube-api-access-hsmb8\") pod \"auth-enabled-test-kserve-5c785cd7f8-g2qxc\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.152599 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.152508    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"]
Sep 16 19:57:41.156763 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.156738    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:41.159437 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.159412    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-zg4pq\""
Sep 16 19:57:41.165109 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.165072    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"]
Sep 16 19:57:41.214156 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.214122    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:41.324284 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.324242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5\" (UID: \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:41.324439 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.324316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkls\" (UniqueName: \"kubernetes.io/projected/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-kube-api-access-wrkls\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5\" (UID: \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:41.425714 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.425634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkls\" (UniqueName: \"kubernetes.io/projected/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-kube-api-access-wrkls\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5\" (UID: \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:41.425714 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.425689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5\" (UID: \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:41.428230 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.428196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5\" (UID: \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:41.434073 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.434051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkls\" (UniqueName: \"kubernetes.io/projected/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-kube-api-access-wrkls\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5\" (UID: \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:41.467872 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.467846    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:41.493428 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.493399    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"]
Sep 16 19:57:41.495502 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:57:41.495475    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5a0328_43a1_45bb_ba27_23dc334f1efd.slice/crio-3d24945ef9c478d698981d9752dd2757b8a87eeed5917f6d22f02a90ce89a121 WatchSource:0}: Error finding container 3d24945ef9c478d698981d9752dd2757b8a87eeed5917f6d22f02a90ce89a121: Status 404 returned error can't find the container with id 3d24945ef9c478d698981d9752dd2757b8a87eeed5917f6d22f02a90ce89a121
Sep 16 19:57:41.748528 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:41.748496    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"]
Sep 16 19:57:41.749488 ip-10-0-136-120 kubenswrapper[2561]: W0916 19:57:41.749463    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8b7bf6_e2c1_4ae6_ab87_1306cf9a7840.slice/crio-7636787af317059b8a554328446edd3e866c18e3ff630655dfbc3f5e4d33d6b4 WatchSource:0}: Error finding container 7636787af317059b8a554328446edd3e866c18e3ff630655dfbc3f5e4d33d6b4: Status 404 returned error can't find the container with id 7636787af317059b8a554328446edd3e866c18e3ff630655dfbc3f5e4d33d6b4
Sep 16 19:57:42.033244 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:42.033201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" event={"ID":"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840","Type":"ContainerStarted","Data":"dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa"}
Sep 16 19:57:42.033718 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:42.033274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" event={"ID":"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840","Type":"ContainerStarted","Data":"7636787af317059b8a554328446edd3e866c18e3ff630655dfbc3f5e4d33d6b4"}
Sep 16 19:57:42.033718 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:42.033308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:57:42.034812 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:42.034786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" event={"ID":"de5a0328-43a1-45bb-ba27-23dc334f1efd","Type":"ContainerStarted","Data":"f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12"}
Sep 16 19:57:42.034924 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:42.034835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" event={"ID":"de5a0328-43a1-45bb-ba27-23dc334f1efd","Type":"ContainerStarted","Data":"3d24945ef9c478d698981d9752dd2757b8a87eeed5917f6d22f02a90ce89a121"}
Sep 16 19:57:42.073720 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:42.073672    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" podStartSLOduration=1.073656865 podStartE2EDuration="1.073656865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:42.049675728 +0000 UTC m=+1108.821546140" watchObservedRunningTime="2026-09-16 19:57:42.073656865 +0000 UTC m=+1108.845527276"
Sep 16 19:57:44.046127 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:44.046038    2561 generic.go:356] "Generic (PLEG): container finished" podID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerID="f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12" exitCode=0
Sep 16 19:57:44.046127 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:44.046109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" event={"ID":"de5a0328-43a1-45bb-ba27-23dc334f1efd","Type":"ContainerDied","Data":"f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12"}
Sep 16 19:57:45.053601 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:45.053568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" event={"ID":"de5a0328-43a1-45bb-ba27-23dc334f1efd","Type":"ContainerStarted","Data":"b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b"}
Sep 16 19:57:45.080231 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:45.080179    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podStartSLOduration=2.879331357 podStartE2EDuration="5.080162456s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.200831099s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:45.078147615 +0000 UTC m=+1111.850018026" watchObservedRunningTime="2026-09-16 19:57:45.080162456 +0000 UTC m=+1111.852032867"
Sep 16 19:57:45.215279 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:45.215244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:51.214675 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:51.214631    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:57:51.216171 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:57:51.216142    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:58:01.214703 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:58:01.214656    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:58:11.214855 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:58:11.214810    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:58:13.042154 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:58:13.042124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 19:58:21.214672 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:58:21.214630    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:58:31.215072 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:58:31.215021    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:58:41.214699 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:58:41.214607    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:58:51.214619 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:58:51.214576    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:59:01.215459 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:01.215405    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:59:11.215327 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:11.215284    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:59:13.671791 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:13.671763    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:59:13.672440 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:13.672418    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 19:59:21.214707 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:21.214660    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:59:31.215073 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:31.215030    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 19:59:41.224650 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:41.224614    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:59:41.232439 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:41.232413    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 19:59:46.232371 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:46.232332    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"]
Sep 16 19:59:46.237041 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:46.237014    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"]
Sep 16 19:59:53.041326 ip-10-0-136-120 kubenswrapper[2561]: I0916 19:59:53.041280    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:01.252101 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:01.252056    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerName="main" containerID="cri-o://dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa" gracePeriod=45
Sep 16 20:00:01.258271 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:01.258220    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main" containerID="cri-o://b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b" gracePeriod=45
Sep 16 20:00:02.007180 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.007153    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 20:00:02.052490 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-kserve-provision-location\") pod \"de5a0328-43a1-45bb-ba27-23dc334f1efd\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") "
Sep 16 20:00:02.052490 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052447    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-dshm\") pod \"de5a0328-43a1-45bb-ba27-23dc334f1efd\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") "
Sep 16 20:00:02.052490 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052469    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-tmp-dir\") pod \"de5a0328-43a1-45bb-ba27-23dc334f1efd\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") "
Sep 16 20:00:02.052490 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052500    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsmb8\" (UniqueName: \"kubernetes.io/projected/de5a0328-43a1-45bb-ba27-23dc334f1efd-kube-api-access-hsmb8\") pod \"de5a0328-43a1-45bb-ba27-23dc334f1efd\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") "
Sep 16 20:00:02.052827 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052533    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-home\") pod \"de5a0328-43a1-45bb-ba27-23dc334f1efd\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") "
Sep 16 20:00:02.052827 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052551    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-model-cache\") pod \"de5a0328-43a1-45bb-ba27-23dc334f1efd\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") "
Sep 16 20:00:02.052827 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052571    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5a0328-43a1-45bb-ba27-23dc334f1efd-tls-certs\") pod \"de5a0328-43a1-45bb-ba27-23dc334f1efd\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") "
Sep 16 20:00:02.052827 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052605    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5a0328-43a1-45bb-ba27-23dc334f1efd-cabundle-cert\") pod \"de5a0328-43a1-45bb-ba27-23dc334f1efd\" (UID: \"de5a0328-43a1-45bb-ba27-23dc334f1efd\") "
Sep 16 20:00:02.053024 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.052865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-model-cache" (OuterVolumeSpecName: "model-cache") pod "de5a0328-43a1-45bb-ba27-23dc334f1efd" (UID: "de5a0328-43a1-45bb-ba27-23dc334f1efd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:02.053101 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.053074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5a0328-43a1-45bb-ba27-23dc334f1efd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de5a0328-43a1-45bb-ba27-23dc334f1efd" (UID: "de5a0328-43a1-45bb-ba27-23dc334f1efd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:00:02.053299 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.053270    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-home" (OuterVolumeSpecName: "home") pod "de5a0328-43a1-45bb-ba27-23dc334f1efd" (UID: "de5a0328-43a1-45bb-ba27-23dc334f1efd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:02.055312 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.055264    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-dshm" (OuterVolumeSpecName: "dshm") pod "de5a0328-43a1-45bb-ba27-23dc334f1efd" (UID: "de5a0328-43a1-45bb-ba27-23dc334f1efd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:02.055535 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.055510    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5a0328-43a1-45bb-ba27-23dc334f1efd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de5a0328-43a1-45bb-ba27-23dc334f1efd" (UID: "de5a0328-43a1-45bb-ba27-23dc334f1efd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:02.055904 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.055876    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5a0328-43a1-45bb-ba27-23dc334f1efd-kube-api-access-hsmb8" (OuterVolumeSpecName: "kube-api-access-hsmb8") pod "de5a0328-43a1-45bb-ba27-23dc334f1efd" (UID: "de5a0328-43a1-45bb-ba27-23dc334f1efd"). InnerVolumeSpecName "kube-api-access-hsmb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:02.065916 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.065879    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de5a0328-43a1-45bb-ba27-23dc334f1efd" (UID: "de5a0328-43a1-45bb-ba27-23dc334f1efd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:02.075117 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.075085    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de5a0328-43a1-45bb-ba27-23dc334f1efd" (UID: "de5a0328-43a1-45bb-ba27-23dc334f1efd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:02.154079 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.154026    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:02.154079 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.154075    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:02.154079 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.154089    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5a0328-43a1-45bb-ba27-23dc334f1efd-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:02.154079 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.154097    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5a0328-43a1-45bb-ba27-23dc334f1efd-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:02.154401 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.154106    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:02.154401 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.154114    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:02.154401 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.154122    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5a0328-43a1-45bb-ba27-23dc334f1efd-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:02.154401 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.154130    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsmb8\" (UniqueName: \"kubernetes.io/projected/de5a0328-43a1-45bb-ba27-23dc334f1efd-kube-api-access-hsmb8\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:02.669170 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.669134    2561 generic.go:356] "Generic (PLEG): container finished" podID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerID="b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b" exitCode=0
Sep 16 20:00:02.669577 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.669193    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"
Sep 16 20:00:02.669577 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.669220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" event={"ID":"de5a0328-43a1-45bb-ba27-23dc334f1efd","Type":"ContainerDied","Data":"b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b"}
Sep 16 20:00:02.669577 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.669281    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc" event={"ID":"de5a0328-43a1-45bb-ba27-23dc334f1efd","Type":"ContainerDied","Data":"3d24945ef9c478d698981d9752dd2757b8a87eeed5917f6d22f02a90ce89a121"}
Sep 16 20:00:02.669577 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.669307    2561 scope.go:117] "RemoveContainer" containerID="b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b"
Sep 16 20:00:02.678637 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.678617    2561 scope.go:117] "RemoveContainer" containerID="f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12"
Sep 16 20:00:02.686758 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.686736    2561 scope.go:117] "RemoveContainer" containerID="b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b"
Sep 16 20:00:02.687035 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:00:02.687018    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b\": container with ID starting with b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b not found: ID does not exist" containerID="b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b"
Sep 16 20:00:02.687077 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.687043    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b"} err="failed to get container status \"b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b\": rpc error: code = NotFound desc = could not find container \"b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b\": container with ID starting with b7b3a1546e2fab7e2f83e061039b47bf3023f2dffb2513215fc63e4aa4fca12b not found: ID does not exist"
Sep 16 20:00:02.687077 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.687059    2561 scope.go:117] "RemoveContainer" containerID="f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12"
Sep 16 20:00:02.687301 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:00:02.687285    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12\": container with ID starting with f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12 not found: ID does not exist" containerID="f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12"
Sep 16 20:00:02.687341 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.687306    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12"} err="failed to get container status \"f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12\": rpc error: code = NotFound desc = could not find container \"f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12\": container with ID starting with f528433db95d6fb8fd1dba3b86212e53e773774785495594f7e04d86818b2f12 not found: ID does not exist"
Sep 16 20:00:02.703071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.703042    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"]
Sep 16 20:00:02.710628 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:02.710601    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c785cd7f8-g2qxc"]
Sep 16 20:00:03.042099 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:03.042066    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:03.682550 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:03.682515    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" path="/var/lib/kubelet/pods/de5a0328-43a1-45bb-ba27-23dc334f1efd/volumes"
Sep 16 20:00:13.042126 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:13.042038    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:13.042505 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:13.042173    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 20:00:23.041862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:23.041821    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:31.492097 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.492067    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 20:00:31.577983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.577900    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-tls-certs\") pod \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\" (UID: \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\") "
Sep 16 20:00:31.577983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.577946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrkls\" (UniqueName: \"kubernetes.io/projected/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-kube-api-access-wrkls\") pod \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\" (UID: \"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840\") "
Sep 16 20:00:31.580047 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.580014    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" (UID: "1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:31.580168 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.580087    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-kube-api-access-wrkls" (OuterVolumeSpecName: "kube-api-access-wrkls") pod "1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" (UID: "1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840"). InnerVolumeSpecName "kube-api-access-wrkls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:31.678662 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.678621    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.678662 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.678653    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkls\" (UniqueName: \"kubernetes.io/projected/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840-kube-api-access-wrkls\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.787152 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.787114    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerID="dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa" exitCode=0
Sep 16 20:00:31.787363 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.787161    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"
Sep 16 20:00:31.787363 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.787202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" event={"ID":"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840","Type":"ContainerDied","Data":"dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa"}
Sep 16 20:00:31.787363 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.787259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5" event={"ID":"1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840","Type":"ContainerDied","Data":"7636787af317059b8a554328446edd3e866c18e3ff630655dfbc3f5e4d33d6b4"}
Sep 16 20:00:31.787363 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.787285    2561 scope.go:117] "RemoveContainer" containerID="dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa"
Sep 16 20:00:31.795270 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.795250    2561 scope.go:117] "RemoveContainer" containerID="dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa"
Sep 16 20:00:31.795569 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:00:31.795547    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa\": container with ID starting with dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa not found: ID does not exist" containerID="dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa"
Sep 16 20:00:31.795664 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.795574    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa"} err="failed to get container status \"dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa\": rpc error: code = NotFound desc = could not find container \"dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa\": container with ID starting with dc6e765181fd82770e92534cf34d7c09b1a99bdfe1e6b865e799f628c7f2c3aa not found: ID does not exist"
Sep 16 20:00:31.812823 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.812796    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"]
Sep 16 20:00:31.823584 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:31.823561    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-wbrd5"]
Sep 16 20:00:33.681790 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:33.681756    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" path="/var/lib/kubelet/pods/1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840/volumes"
Sep 16 20:00:43.745300 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745255    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"]
Sep 16 20:00:43.745689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745533    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="storage-initializer"
Sep 16 20:00:43.745689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745545    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="storage-initializer"
Sep 16 20:00:43.745689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745554    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main"
Sep 16 20:00:43.745689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745559    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main"
Sep 16 20:00:43.745689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745576    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerName="main"
Sep 16 20:00:43.745689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745582    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerName="main"
Sep 16 20:00:43.745689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745631    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8b7bf6-e2c1-4ae6-ab87-1306cf9a7840" containerName="main"
Sep 16 20:00:43.745689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.745638    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5a0328-43a1-45bb-ba27-23dc334f1efd" containerName="main"
Sep 16 20:00:43.748625 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.748602    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.755133 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.755110    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:00:43.755133 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.755120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 20:00:43.755305 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.755222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 20:00:43.756445 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.756420    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 20:00:43.756523 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.756420    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-kr4bm\""
Sep 16 20:00:43.756572 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.756470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 20:00:43.771429 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.771404    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"]
Sep 16 20:00:43.863933 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.863902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a24e96-356d-4158-97c9-04ed0d91631c-tls-certs\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.864113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.863940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-model-cache\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.864113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.863960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-tmp-dir\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.864113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.864012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06a24e96-356d-4158-97c9-04ed0d91631c-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.864113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.864051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.864113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.864069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwrs\" (UniqueName: \"kubernetes.io/projected/06a24e96-356d-4158-97c9-04ed0d91631c-kube-api-access-dhwrs\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.864113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.864093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-home\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.864113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.864113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-dshm\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.964609 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.964563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a24e96-356d-4158-97c9-04ed0d91631c-tls-certs\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.964609 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.964611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-model-cache\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.964853 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.964626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-tmp-dir\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.964853 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.964677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06a24e96-356d-4158-97c9-04ed0d91631c-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.964853 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.964711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.964853 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.964730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwrs\" (UniqueName: \"kubernetes.io/projected/06a24e96-356d-4158-97c9-04ed0d91631c-kube-api-access-dhwrs\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.964853 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.964762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-home\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.964853 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.964810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-dshm\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.965123 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.965042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-model-cache\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.965123 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.965060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-tmp-dir\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.965198 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.965184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-home\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.965252 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.965244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.965514 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.965489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06a24e96-356d-4158-97c9-04ed0d91631c-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.967051 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.967029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-dshm\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.967144 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.967125    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a24e96-356d-4158-97c9-04ed0d91631c-tls-certs\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:43.982229 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:43.982204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwrs\" (UniqueName: \"kubernetes.io/projected/06a24e96-356d-4158-97c9-04ed0d91631c-kube-api-access-dhwrs\") pod \"auth-invalid-token-test-kserve-6dc94487bb-m8rpp\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:44.058749 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.058715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:44.096697 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.096664    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"]
Sep 16 20:00:44.100010 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.099983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.104101 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.104079    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-g952s\""
Sep 16 20:00:44.113896 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.113872    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"]
Sep 16 20:00:44.166769 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.166736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30373e6b-b631-4d88-9b7d-45da864a3815-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp\" (UID: \"30373e6b-b631-4d88-9b7d-45da864a3815\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.166930 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.166782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtc8\" (UniqueName: \"kubernetes.io/projected/30373e6b-b631-4d88-9b7d-45da864a3815-kube-api-access-2qtc8\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp\" (UID: \"30373e6b-b631-4d88-9b7d-45da864a3815\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.268045 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.268013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30373e6b-b631-4d88-9b7d-45da864a3815-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp\" (UID: \"30373e6b-b631-4d88-9b7d-45da864a3815\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.268192 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.268056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtc8\" (UniqueName: \"kubernetes.io/projected/30373e6b-b631-4d88-9b7d-45da864a3815-kube-api-access-2qtc8\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp\" (UID: \"30373e6b-b631-4d88-9b7d-45da864a3815\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.270650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.270627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30373e6b-b631-4d88-9b7d-45da864a3815-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp\" (UID: \"30373e6b-b631-4d88-9b7d-45da864a3815\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.284133 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.284104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtc8\" (UniqueName: \"kubernetes.io/projected/30373e6b-b631-4d88-9b7d-45da864a3815-kube-api-access-2qtc8\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp\" (UID: \"30373e6b-b631-4d88-9b7d-45da864a3815\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.348971 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.348888    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"]
Sep 16 20:00:44.355955 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:00:44.355926    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a24e96_356d_4158_97c9_04ed0d91631c.slice/crio-1355e652ac70484f1b7b77ae805ffe5c10815d2fde00627bfe24c537e4dc6876 WatchSource:0}: Error finding container 1355e652ac70484f1b7b77ae805ffe5c10815d2fde00627bfe24c537e4dc6876: Status 404 returned error can't find the container with id 1355e652ac70484f1b7b77ae805ffe5c10815d2fde00627bfe24c537e4dc6876
Sep 16 20:00:44.411977 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.411955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.701946 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.701915    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"]
Sep 16 20:00:44.704132 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:00:44.704102    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30373e6b_b631_4d88_9b7d_45da864a3815.slice/crio-1a59e7b6c7b1a62e381adcbc174e904eec02d90049abc01ea6635f7847b23c37 WatchSource:0}: Error finding container 1a59e7b6c7b1a62e381adcbc174e904eec02d90049abc01ea6635f7847b23c37: Status 404 returned error can't find the container with id 1a59e7b6c7b1a62e381adcbc174e904eec02d90049abc01ea6635f7847b23c37
Sep 16 20:00:44.843349 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.843308    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" event={"ID":"06a24e96-356d-4158-97c9-04ed0d91631c","Type":"ContainerStarted","Data":"946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf"}
Sep 16 20:00:44.843824 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.843403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" event={"ID":"06a24e96-356d-4158-97c9-04ed0d91631c","Type":"ContainerStarted","Data":"1355e652ac70484f1b7b77ae805ffe5c10815d2fde00627bfe24c537e4dc6876"}
Sep 16 20:00:44.845124 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.845090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" event={"ID":"30373e6b-b631-4d88-9b7d-45da864a3815","Type":"ContainerStarted","Data":"3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f"}
Sep 16 20:00:44.845259 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.845146    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" event={"ID":"30373e6b-b631-4d88-9b7d-45da864a3815","Type":"ContainerStarted","Data":"1a59e7b6c7b1a62e381adcbc174e904eec02d90049abc01ea6635f7847b23c37"}
Sep 16 20:00:44.845323 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.845264    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:00:44.889039 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:44.888977    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" podStartSLOduration=0.888957436 podStartE2EDuration="888.957436ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:00:44.888356715 +0000 UTC m=+1291.660227149" watchObservedRunningTime="2026-09-16 20:00:44.888957436 +0000 UTC m=+1291.660827849"
Sep 16 20:00:46.855161 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:46.855121    2561 generic.go:356] "Generic (PLEG): container finished" podID="06a24e96-356d-4158-97c9-04ed0d91631c" containerID="946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf" exitCode=0
Sep 16 20:00:46.855556 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:46.855197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" event={"ID":"06a24e96-356d-4158-97c9-04ed0d91631c","Type":"ContainerDied","Data":"946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf"}
Sep 16 20:00:47.861423 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:47.861385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" event={"ID":"06a24e96-356d-4158-97c9-04ed0d91631c","Type":"ContainerStarted","Data":"d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604"}
Sep 16 20:00:47.888685 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:47.888630    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podStartSLOduration=2.640634059 podStartE2EDuration="4.888615786s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.247981727s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:00:47.886965542 +0000 UTC m=+1294.658835953" watchObservedRunningTime="2026-09-16 20:00:47.888615786 +0000 UTC m=+1294.660486196"
Sep 16 20:00:48.059669 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:48.059622    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:54.059464 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:54.059420    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:00:54.060789 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:00:54.060759    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:01:04.060007 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:04.059954    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:01:14.059716 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:14.059668    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:01:15.851512 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:15.851477    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:01:24.060006 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:24.059960    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:01:34.059181 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:34.059134    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:01:44.059990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:44.059891    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:01:47.389389 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.389334    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"]
Sep 16 20:01:47.392238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.392212    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:47.394799 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.394767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 20:01:47.395985 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.395964    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-k5jxh\""
Sep 16 20:01:47.396105 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.396041    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 20:01:47.396105 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.396051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-c9sjl\""
Sep 16 20:01:47.396222 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.396176    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 20:01:47.406779 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.406755    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"]
Sep 16 20:01:47.471035 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.470999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmpf\" (UniqueName: \"kubernetes.io/projected/e26e961d-d948-4114-b6c5-f0923b608add-kube-api-access-kpmpf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn\" (UID: \"e26e961d-d948-4114-b6c5-f0923b608add\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:47.471208 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.471071    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26e961d-d948-4114-b6c5-f0923b608add-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn\" (UID: \"e26e961d-d948-4114-b6c5-f0923b608add\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:47.572452 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.572418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26e961d-d948-4114-b6c5-f0923b608add-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn\" (UID: \"e26e961d-d948-4114-b6c5-f0923b608add\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:47.572636 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.572491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmpf\" (UniqueName: \"kubernetes.io/projected/e26e961d-d948-4114-b6c5-f0923b608add-kube-api-access-kpmpf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn\" (UID: \"e26e961d-d948-4114-b6c5-f0923b608add\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:47.574915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.574886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26e961d-d948-4114-b6c5-f0923b608add-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn\" (UID: \"e26e961d-d948-4114-b6c5-f0923b608add\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:47.581424 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.581396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmpf\" (UniqueName: \"kubernetes.io/projected/e26e961d-d948-4114-b6c5-f0923b608add-kube-api-access-kpmpf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn\" (UID: \"e26e961d-d948-4114-b6c5-f0923b608add\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:47.702796 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.702712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:47.979699 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:47.979535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"]
Sep 16 20:01:47.982299 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:01:47.982265    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26e961d_d948_4114_b6c5_f0923b608add.slice/crio-1ab45264dd7e28023a222b91320eb80fd545901a8136bcf0d5e5535478ed7ded WatchSource:0}: Error finding container 1ab45264dd7e28023a222b91320eb80fd545901a8136bcf0d5e5535478ed7ded: Status 404 returned error can't find the container with id 1ab45264dd7e28023a222b91320eb80fd545901a8136bcf0d5e5535478ed7ded
Sep 16 20:01:48.121342 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:48.121296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" event={"ID":"e26e961d-d948-4114-b6c5-f0923b608add","Type":"ContainerStarted","Data":"8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa"}
Sep 16 20:01:48.121576 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:48.121386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" event={"ID":"e26e961d-d948-4114-b6c5-f0923b608add","Type":"ContainerStarted","Data":"1ab45264dd7e28023a222b91320eb80fd545901a8136bcf0d5e5535478ed7ded"}
Sep 16 20:01:48.121576 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:48.121514    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:01:48.140604 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:48.140510    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" podStartSLOduration=1.140490565 podStartE2EDuration="1.140490565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:48.139194275 +0000 UTC m=+1354.911064681" watchObservedRunningTime="2026-09-16 20:01:48.140490565 +0000 UTC m=+1354.912360976"
Sep 16 20:01:54.059951 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:01:54.059914    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:02:04.059962 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:04.059913    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:02:14.060107 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:14.060062    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:02:19.128927 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:19.128887    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:21.058014 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:21.057979    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"]
Sep 16 20:02:21.059857 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:21.059818    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:24.059623 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:24.059575    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:02:31.059980 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:31.059938    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:34.059641 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:34.059600    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 16 20:02:36.076719 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:36.076676    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main" containerID="cri-o://8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa" gracePeriod=45
Sep 16 20:02:41.060137 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:41.060096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:44.069863 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:44.069824    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:02:44.078122 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:44.078096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:02:46.220059 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:46.220025    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"]
Sep 16 20:02:46.223445 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:46.223411    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"]
Sep 16 20:02:51.060130 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:51.060085    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:55.852069 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:02:55.852028    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:01.060606 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:01.060566    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:01.242628 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:01.242581    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" containerName="main" containerID="cri-o://3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f" gracePeriod=45
Sep 16 20:03:01.243560 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:01.243534    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main" containerID="cri-o://d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604" gracePeriod=45
Sep 16 20:03:01.998838 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:01.998808    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:03:02.079686 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079600    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-model-cache\") pod \"06a24e96-356d-4158-97c9-04ed0d91631c\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") "
Sep 16 20:03:02.079686 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079647    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a24e96-356d-4158-97c9-04ed0d91631c-tls-certs\") pod \"06a24e96-356d-4158-97c9-04ed0d91631c\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") "
Sep 16 20:03:02.079686 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079672    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-tmp-dir\") pod \"06a24e96-356d-4158-97c9-04ed0d91631c\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") "
Sep 16 20:03:02.080219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-home\") pod \"06a24e96-356d-4158-97c9-04ed0d91631c\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") "
Sep 16 20:03:02.080219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079735    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-dshm\") pod \"06a24e96-356d-4158-97c9-04ed0d91631c\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") "
Sep 16 20:03:02.080219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079783    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06a24e96-356d-4158-97c9-04ed0d91631c-cabundle-cert\") pod \"06a24e96-356d-4158-97c9-04ed0d91631c\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") "
Sep 16 20:03:02.080219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhwrs\" (UniqueName: \"kubernetes.io/projected/06a24e96-356d-4158-97c9-04ed0d91631c-kube-api-access-dhwrs\") pod \"06a24e96-356d-4158-97c9-04ed0d91631c\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") "
Sep 16 20:03:02.080219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079866    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-kserve-provision-location\") pod \"06a24e96-356d-4158-97c9-04ed0d91631c\" (UID: \"06a24e96-356d-4158-97c9-04ed0d91631c\") "
Sep 16 20:03:02.080219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.079943    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-model-cache" (OuterVolumeSpecName: "model-cache") pod "06a24e96-356d-4158-97c9-04ed0d91631c" (UID: "06a24e96-356d-4158-97c9-04ed0d91631c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:02.080219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.080057    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.080620 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.080326    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a24e96-356d-4158-97c9-04ed0d91631c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06a24e96-356d-4158-97c9-04ed0d91631c" (UID: "06a24e96-356d-4158-97c9-04ed0d91631c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:03:02.080620 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.080535    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-home" (OuterVolumeSpecName: "home") pod "06a24e96-356d-4158-97c9-04ed0d91631c" (UID: "06a24e96-356d-4158-97c9-04ed0d91631c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:02.082077 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.082053    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a24e96-356d-4158-97c9-04ed0d91631c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06a24e96-356d-4158-97c9-04ed0d91631c" (UID: "06a24e96-356d-4158-97c9-04ed0d91631c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:02.082387 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.082357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-dshm" (OuterVolumeSpecName: "dshm") pod "06a24e96-356d-4158-97c9-04ed0d91631c" (UID: "06a24e96-356d-4158-97c9-04ed0d91631c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:02.082576 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.082561    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a24e96-356d-4158-97c9-04ed0d91631c-kube-api-access-dhwrs" (OuterVolumeSpecName: "kube-api-access-dhwrs") pod "06a24e96-356d-4158-97c9-04ed0d91631c" (UID: "06a24e96-356d-4158-97c9-04ed0d91631c"). InnerVolumeSpecName "kube-api-access-dhwrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:02.093169 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.093135    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06a24e96-356d-4158-97c9-04ed0d91631c" (UID: "06a24e96-356d-4158-97c9-04ed0d91631c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:02.103479 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.103432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06a24e96-356d-4158-97c9-04ed0d91631c" (UID: "06a24e96-356d-4158-97c9-04ed0d91631c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:02.180865 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.180825    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.180865 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.180859    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a24e96-356d-4158-97c9-04ed0d91631c-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.180865 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.180869    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.180865 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.180877    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.181119 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.180885    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06a24e96-356d-4158-97c9-04ed0d91631c-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.181119 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.180892    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06a24e96-356d-4158-97c9-04ed0d91631c-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.181119 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.180900    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwrs\" (UniqueName: \"kubernetes.io/projected/06a24e96-356d-4158-97c9-04ed0d91631c-kube-api-access-dhwrs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.454715 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.454623    2561 generic.go:356] "Generic (PLEG): container finished" podID="06a24e96-356d-4158-97c9-04ed0d91631c" containerID="d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604" exitCode=0
Sep 16 20:03:02.454715 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.454683    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"
Sep 16 20:03:02.454893 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.454714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" event={"ID":"06a24e96-356d-4158-97c9-04ed0d91631c","Type":"ContainerDied","Data":"d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604"}
Sep 16 20:03:02.454893 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.454795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp" event={"ID":"06a24e96-356d-4158-97c9-04ed0d91631c","Type":"ContainerDied","Data":"1355e652ac70484f1b7b77ae805ffe5c10815d2fde00627bfe24c537e4dc6876"}
Sep 16 20:03:02.454893 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.454820    2561 scope.go:117] "RemoveContainer" containerID="d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604"
Sep 16 20:03:02.463425 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.463405    2561 scope.go:117] "RemoveContainer" containerID="946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf"
Sep 16 20:03:02.470619 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.470603    2561 scope.go:117] "RemoveContainer" containerID="d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604"
Sep 16 20:03:02.470904 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:03:02.470881    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604\": container with ID starting with d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604 not found: ID does not exist" containerID="d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604"
Sep 16 20:03:02.470969 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.470910    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604"} err="failed to get container status \"d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604\": rpc error: code = NotFound desc = could not find container \"d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604\": container with ID starting with d1ab5ee0cbf96ee92f4c0c637aeaf5733bbe677b7f1c89cc3df0b8de472b4604 not found: ID does not exist"
Sep 16 20:03:02.470969 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.470926    2561 scope.go:117] "RemoveContainer" containerID="946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf"
Sep 16 20:03:02.471186 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:03:02.471168    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf\": container with ID starting with 946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf not found: ID does not exist" containerID="946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf"
Sep 16 20:03:02.471228 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.471195    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf"} err="failed to get container status \"946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf\": rpc error: code = NotFound desc = could not find container \"946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf\": container with ID starting with 946ba99a4b04fcb9e200c19b4f3e167190382c0792144d4418f862d42dd767cf not found: ID does not exist"
Sep 16 20:03:02.478565 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.478469    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"]
Sep 16 20:03:02.481594 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:02.481571    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6dc94487bb-m8rpp"]
Sep 16 20:03:03.683805 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:03.683774    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" path="/var/lib/kubelet/pods/06a24e96-356d-4158-97c9-04ed0d91631c/volumes"
Sep 16 20:03:05.851531 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:05.851497    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:06.317007 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.316982    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:03:06.413960 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.413920    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26e961d-d948-4114-b6c5-f0923b608add-tls-certs\") pod \"e26e961d-d948-4114-b6c5-f0923b608add\" (UID: \"e26e961d-d948-4114-b6c5-f0923b608add\") "
Sep 16 20:03:06.414123 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.414024    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpmpf\" (UniqueName: \"kubernetes.io/projected/e26e961d-d948-4114-b6c5-f0923b608add-kube-api-access-kpmpf\") pod \"e26e961d-d948-4114-b6c5-f0923b608add\" (UID: \"e26e961d-d948-4114-b6c5-f0923b608add\") "
Sep 16 20:03:06.416166 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.416134    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26e961d-d948-4114-b6c5-f0923b608add-kube-api-access-kpmpf" (OuterVolumeSpecName: "kube-api-access-kpmpf") pod "e26e961d-d948-4114-b6c5-f0923b608add" (UID: "e26e961d-d948-4114-b6c5-f0923b608add"). InnerVolumeSpecName "kube-api-access-kpmpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:06.416272 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.416181    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26e961d-d948-4114-b6c5-f0923b608add-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e26e961d-d948-4114-b6c5-f0923b608add" (UID: "e26e961d-d948-4114-b6c5-f0923b608add"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:06.475099 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.475007    2561 generic.go:356] "Generic (PLEG): container finished" podID="e26e961d-d948-4114-b6c5-f0923b608add" containerID="8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa" exitCode=0
Sep 16 20:03:06.475099 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.475052    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" event={"ID":"e26e961d-d948-4114-b6c5-f0923b608add","Type":"ContainerDied","Data":"8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa"}
Sep 16 20:03:06.475099 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.475059    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"
Sep 16 20:03:06.475099 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.475095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn" event={"ID":"e26e961d-d948-4114-b6c5-f0923b608add","Type":"ContainerDied","Data":"1ab45264dd7e28023a222b91320eb80fd545901a8136bcf0d5e5535478ed7ded"}
Sep 16 20:03:06.475420 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.475117    2561 scope.go:117] "RemoveContainer" containerID="8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa"
Sep 16 20:03:06.483437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.483419    2561 scope.go:117] "RemoveContainer" containerID="8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa"
Sep 16 20:03:06.483692 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:03:06.483675    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa\": container with ID starting with 8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa not found: ID does not exist" containerID="8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa"
Sep 16 20:03:06.483741 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.483697    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa"} err="failed to get container status \"8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa\": rpc error: code = NotFound desc = could not find container \"8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa\": container with ID starting with 8c6b225fdcd2c008b9bbcc89ec70264c715f4ae3369e558c04286221fa8fecaa not found: ID does not exist"
Sep 16 20:03:06.496643 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.496618    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"]
Sep 16 20:03:06.501089 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.501066    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67lvtrn"]
Sep 16 20:03:06.514676 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.514647    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpmpf\" (UniqueName: \"kubernetes.io/projected/e26e961d-d948-4114-b6c5-f0923b608add-kube-api-access-kpmpf\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:06.514676 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:06.514677    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26e961d-d948-4114-b6c5-f0923b608add-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:07.682500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:07.682467    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26e961d-d948-4114-b6c5-f0923b608add" path="/var/lib/kubelet/pods/e26e961d-d948-4114-b6c5-f0923b608add/volumes"
Sep 16 20:03:15.852007 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:15.851959    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:15.852481 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:15.852086    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:03:25.851181 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:25.851143    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:31.484278 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.484251    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:03:31.589461 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.589357    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qtc8\" (UniqueName: \"kubernetes.io/projected/30373e6b-b631-4d88-9b7d-45da864a3815-kube-api-access-2qtc8\") pod \"30373e6b-b631-4d88-9b7d-45da864a3815\" (UID: \"30373e6b-b631-4d88-9b7d-45da864a3815\") "
Sep 16 20:03:31.589461 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.589420    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30373e6b-b631-4d88-9b7d-45da864a3815-tls-certs\") pod \"30373e6b-b631-4d88-9b7d-45da864a3815\" (UID: \"30373e6b-b631-4d88-9b7d-45da864a3815\") "
Sep 16 20:03:31.590504 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.590467    2561 generic.go:356] "Generic (PLEG): container finished" podID="30373e6b-b631-4d88-9b7d-45da864a3815" containerID="3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f" exitCode=0
Sep 16 20:03:31.590635 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.590512    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"
Sep 16 20:03:31.590635 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.590518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" event={"ID":"30373e6b-b631-4d88-9b7d-45da864a3815","Type":"ContainerDied","Data":"3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f"}
Sep 16 20:03:31.590635 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.590573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp" event={"ID":"30373e6b-b631-4d88-9b7d-45da864a3815","Type":"ContainerDied","Data":"1a59e7b6c7b1a62e381adcbc174e904eec02d90049abc01ea6635f7847b23c37"}
Sep 16 20:03:31.590635 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.590606    2561 scope.go:117] "RemoveContainer" containerID="3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f"
Sep 16 20:03:31.591605 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.591582    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30373e6b-b631-4d88-9b7d-45da864a3815-kube-api-access-2qtc8" (OuterVolumeSpecName: "kube-api-access-2qtc8") pod "30373e6b-b631-4d88-9b7d-45da864a3815" (UID: "30373e6b-b631-4d88-9b7d-45da864a3815"). InnerVolumeSpecName "kube-api-access-2qtc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:31.591837 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.591765    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30373e6b-b631-4d88-9b7d-45da864a3815-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30373e6b-b631-4d88-9b7d-45da864a3815" (UID: "30373e6b-b631-4d88-9b7d-45da864a3815"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:31.605035 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.605014    2561 scope.go:117] "RemoveContainer" containerID="3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f"
Sep 16 20:03:31.605320 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:03:31.605302    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f\": container with ID starting with 3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f not found: ID does not exist" containerID="3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f"
Sep 16 20:03:31.605407 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.605333    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f"} err="failed to get container status \"3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f\": rpc error: code = NotFound desc = could not find container \"3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f\": container with ID starting with 3018262da54a47500a8f5f5e2e9dcdf453a8bbf5299814af75fc46130bf5534f not found: ID does not exist"
Sep 16 20:03:31.690491 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.690467    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtc8\" (UniqueName: \"kubernetes.io/projected/30373e6b-b631-4d88-9b7d-45da864a3815-kube-api-access-2qtc8\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:31.690491 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.690491    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30373e6b-b631-4d88-9b7d-45da864a3815-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:03:31.907944 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.907870    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"]
Sep 16 20:03:31.911729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:31.911705    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc465gfnbp"]
Sep 16 20:03:33.683102 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:33.683069    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" path="/var/lib/kubelet/pods/30373e6b-b631-4d88-9b7d-45da864a3815/volumes"
Sep 16 20:03:48.178392 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178337    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"]
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178650    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178663    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178681    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178686    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178695    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="storage-initializer"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178702    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="storage-initializer"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178711    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" containerName="main"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178719    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" containerName="main"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178769    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26e961d-d948-4114-b6c5-f0923b608add" containerName="main"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178776    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="30373e6b-b631-4d88-9b7d-45da864a3815" containerName="main"
Sep 16 20:03:48.178990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.178781    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="06a24e96-356d-4158-97c9-04ed0d91631c" containerName="main"
Sep 16 20:03:48.181702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.181679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.184521 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.184495    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:03:48.184645 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.184529    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 20:03:48.184741 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.184696    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-x5h6p\""
Sep 16 20:03:48.184741 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.184733    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 20:03:48.184875 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.184800    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 20:03:48.185007 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.184988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 20:03:48.196781 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.196745    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"]
Sep 16 20:03:48.210563 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.210533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tmp-dir\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.210700 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.210608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a83dfc-0078-473a-b9bd-de9efc9a9633-cabundle-cert\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.210700 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.210628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-dshm\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.210700 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.210649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kserve-provision-location\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.210700 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.210673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5rh\" (UniqueName: \"kubernetes.io/projected/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kube-api-access-5x5rh\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.210889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.210716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tls-certs\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.210889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.210734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-home\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.210889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.210796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-model-cache\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.311445 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.311404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a83dfc-0078-473a-b9bd-de9efc9a9633-cabundle-cert\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.311445 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.311445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-dshm\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.311699 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.311464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kserve-provision-location\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.311699 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.311496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5rh\" (UniqueName: \"kubernetes.io/projected/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kube-api-access-5x5rh\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.311699 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.311522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tls-certs\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.311699 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.311540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-home\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.311699 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.311562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-model-cache\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.311699 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.311596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tmp-dir\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.312094 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.312056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kserve-provision-location\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.312188 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.312072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tmp-dir\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.312188 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.312145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-home\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.312188 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.312169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-model-cache\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.312323 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.312228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a83dfc-0078-473a-b9bd-de9efc9a9633-cabundle-cert\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.314371 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.314345    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-dshm\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.314548 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.314533    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tls-certs\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.320072 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.320047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5rh\" (UniqueName: \"kubernetes.io/projected/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kube-api-access-5x5rh\") pod \"auth-disabled-test-kserve-5c9767b895-xtxtt\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.388732 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.388699    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"]
Sep 16 20:03:48.393243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.393219    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:48.396064 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.396039    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-kmkb4\""
Sep 16 20:03:48.401865 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.401838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"]
Sep 16 20:03:48.412166 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.412135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx7lg\" (UniqueName: \"kubernetes.io/projected/09df674c-3745-4af5-9c67-cefda26542b2-kube-api-access-zx7lg\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h\" (UID: \"09df674c-3745-4af5-9c67-cefda26542b2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:48.412279 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.412183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09df674c-3745-4af5-9c67-cefda26542b2-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h\" (UID: \"09df674c-3745-4af5-9c67-cefda26542b2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:48.491525 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.491435    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:48.512705 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.512677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09df674c-3745-4af5-9c67-cefda26542b2-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h\" (UID: \"09df674c-3745-4af5-9c67-cefda26542b2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:48.512828 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.512746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx7lg\" (UniqueName: \"kubernetes.io/projected/09df674c-3745-4af5-9c67-cefda26542b2-kube-api-access-zx7lg\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h\" (UID: \"09df674c-3745-4af5-9c67-cefda26542b2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:48.515059 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.515037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09df674c-3745-4af5-9c67-cefda26542b2-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h\" (UID: \"09df674c-3745-4af5-9c67-cefda26542b2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:48.522408 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.522368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx7lg\" (UniqueName: \"kubernetes.io/projected/09df674c-3745-4af5-9c67-cefda26542b2-kube-api-access-zx7lg\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h\" (UID: \"09df674c-3745-4af5-9c67-cefda26542b2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:48.704297 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.704256    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:48.772479 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.772451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"]
Sep 16 20:03:48.774341 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:03:48.774312    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a83dfc_0078_473a_b9bd_de9efc9a9633.slice/crio-c38a85a20d773928170679f494dcf1e5c2e456d4bcc6c5a6d775bc06ce6a38c6 WatchSource:0}: Error finding container c38a85a20d773928170679f494dcf1e5c2e456d4bcc6c5a6d775bc06ce6a38c6: Status 404 returned error can't find the container with id c38a85a20d773928170679f494dcf1e5c2e456d4bcc6c5a6d775bc06ce6a38c6
Sep 16 20:03:48.981755 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:48.981729    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"]
Sep 16 20:03:48.983920 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:03:48.983893    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09df674c_3745_4af5_9c67_cefda26542b2.slice/crio-a89b4f480178c2c573065767964ab9ac1c5acd7f65c8c3ab8dff752441fd1ab6 WatchSource:0}: Error finding container a89b4f480178c2c573065767964ab9ac1c5acd7f65c8c3ab8dff752441fd1ab6: Status 404 returned error can't find the container with id a89b4f480178c2c573065767964ab9ac1c5acd7f65c8c3ab8dff752441fd1ab6
Sep 16 20:03:49.663905 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:49.663864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" event={"ID":"d9a83dfc-0078-473a-b9bd-de9efc9a9633","Type":"ContainerStarted","Data":"5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b"}
Sep 16 20:03:49.664419 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:49.663961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" event={"ID":"d9a83dfc-0078-473a-b9bd-de9efc9a9633","Type":"ContainerStarted","Data":"c38a85a20d773928170679f494dcf1e5c2e456d4bcc6c5a6d775bc06ce6a38c6"}
Sep 16 20:03:49.665509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:49.665471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" event={"ID":"09df674c-3745-4af5-9c67-cefda26542b2","Type":"ContainerStarted","Data":"fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48"}
Sep 16 20:03:49.665634 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:49.665530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" event={"ID":"09df674c-3745-4af5-9c67-cefda26542b2","Type":"ContainerStarted","Data":"a89b4f480178c2c573065767964ab9ac1c5acd7f65c8c3ab8dff752441fd1ab6"}
Sep 16 20:03:49.665634 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:49.665592    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:03:49.709317 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:49.709262    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" podStartSLOduration=1.7092467949999999 podStartE2EDuration="1.709246795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:03:49.707607585 +0000 UTC m=+1476.479477996" watchObservedRunningTime="2026-09-16 20:03:49.709246795 +0000 UTC m=+1476.481117255"
Sep 16 20:03:51.675984 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:51.675948    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerID="5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b" exitCode=0
Sep 16 20:03:51.676360 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:51.676023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" event={"ID":"d9a83dfc-0078-473a-b9bd-de9efc9a9633","Type":"ContainerDied","Data":"5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b"}
Sep 16 20:03:52.683002 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:52.682961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" event={"ID":"d9a83dfc-0078-473a-b9bd-de9efc9a9633","Type":"ContainerStarted","Data":"c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d"}
Sep 16 20:03:52.710201 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:52.710148    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podStartSLOduration=2.4436425760000002 podStartE2EDuration="4.710131627s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.266489051s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:03:52.708884415 +0000 UTC m=+1479.480754854" watchObservedRunningTime="2026-09-16 20:03:52.710131627 +0000 UTC m=+1479.482002038"
Sep 16 20:03:53.492112 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:53.492073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:58.492031 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:58.491988    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:03:58.493546 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:03:58.493518    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:04:08.492261 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:08.492211    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:04:13.701064 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:13.701027    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:04:13.701723 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:13.701701    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:04:18.492258 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:18.492204    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:04:20.672302 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:20.672273    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:04:28.492935 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:28.492857    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:04:38.492323 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:38.492224    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:04:48.492199 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:48.492154    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:04:58.492968 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:04:58.492922    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:05:08.492772 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:05:08.492724    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:05:18.492040 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:05:18.491994    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:05:28.491889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:05:28.491845    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:05:38.492910 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:05:38.492862    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 16 20:05:48.502649 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:05:48.502609    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:05:48.510685 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:05:48.510645    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:05:58.491736 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:05:58.491703    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"]
Sep 16 20:05:58.496753 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:05:58.496718    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"]
Sep 16 20:06:00.672002 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:00.671961    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:10.672366 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:10.672275    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:13.517229 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:13.517155    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main" containerID="cri-o://fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48" gracePeriod=45
Sep 16 20:06:13.521346 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:13.521310    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main" containerID="cri-o://c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d" gracePeriod=45
Sep 16 20:06:14.264047 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.264022    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:06:14.293425 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.293394    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerID="c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d" exitCode=0
Sep 16 20:06:14.293574 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.293456    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"
Sep 16 20:06:14.293574 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.293471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" event={"ID":"d9a83dfc-0078-473a-b9bd-de9efc9a9633","Type":"ContainerDied","Data":"c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d"}
Sep 16 20:06:14.293574 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.293530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt" event={"ID":"d9a83dfc-0078-473a-b9bd-de9efc9a9633","Type":"ContainerDied","Data":"c38a85a20d773928170679f494dcf1e5c2e456d4bcc6c5a6d775bc06ce6a38c6"}
Sep 16 20:06:14.293574 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.293559    2561 scope.go:117] "RemoveContainer" containerID="c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d"
Sep 16 20:06:14.301416 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.301398    2561 scope.go:117] "RemoveContainer" containerID="5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b"
Sep 16 20:06:14.310804 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.310775    2561 scope.go:117] "RemoveContainer" containerID="c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d"
Sep 16 20:06:14.311106 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:06:14.311084    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d\": container with ID starting with c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d not found: ID does not exist" containerID="c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d"
Sep 16 20:06:14.311187 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.311113    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d"} err="failed to get container status \"c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d\": rpc error: code = NotFound desc = could not find container \"c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d\": container with ID starting with c4b28d7c68a46039c9f0d33cf90b2fa3fd1bbbb399a2d27a65fe631af92fea6d not found: ID does not exist"
Sep 16 20:06:14.311187 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.311130    2561 scope.go:117] "RemoveContainer" containerID="5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b"
Sep 16 20:06:14.311419 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:06:14.311400    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b\": container with ID starting with 5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b not found: ID does not exist" containerID="5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b"
Sep 16 20:06:14.311502 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.311424    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b"} err="failed to get container status \"5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b\": rpc error: code = NotFound desc = could not find container \"5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b\": container with ID starting with 5e8563faa15a00b312d4447f74b4c2963029ad6b05be9a8228c8f7eb3338ac2b not found: ID does not exist"
Sep 16 20:06:14.312598 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.312576    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tls-certs\") pod \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") "
Sep 16 20:06:14.312715 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.312614    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kserve-provision-location\") pod \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") "
Sep 16 20:06:14.312715 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.312649    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-dshm\") pod \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") "
Sep 16 20:06:14.312715 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.312665    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a83dfc-0078-473a-b9bd-de9efc9a9633-cabundle-cert\") pod \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") "
Sep 16 20:06:14.312715 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.312684    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-model-cache\") pod \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") "
Sep 16 20:06:14.312715 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.312711    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tmp-dir\") pod \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") "
Sep 16 20:06:14.313509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.312742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x5rh\" (UniqueName: \"kubernetes.io/projected/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kube-api-access-5x5rh\") pod \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") "
Sep 16 20:06:14.313509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.312778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-home\") pod \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\" (UID: \"d9a83dfc-0078-473a-b9bd-de9efc9a9633\") "
Sep 16 20:06:14.313509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.313029    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9a83dfc-0078-473a-b9bd-de9efc9a9633" (UID: "d9a83dfc-0078-473a-b9bd-de9efc9a9633"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:14.313509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.313157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a83dfc-0078-473a-b9bd-de9efc9a9633-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9a83dfc-0078-473a-b9bd-de9efc9a9633" (UID: "d9a83dfc-0078-473a-b9bd-de9efc9a9633"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:06:14.313728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.313509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-home" (OuterVolumeSpecName: "home") pod "d9a83dfc-0078-473a-b9bd-de9efc9a9633" (UID: "d9a83dfc-0078-473a-b9bd-de9efc9a9633"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:14.315396 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.315352    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9a83dfc-0078-473a-b9bd-de9efc9a9633" (UID: "d9a83dfc-0078-473a-b9bd-de9efc9a9633"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:06:14.315499 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.315447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kube-api-access-5x5rh" (OuterVolumeSpecName: "kube-api-access-5x5rh") pod "d9a83dfc-0078-473a-b9bd-de9efc9a9633" (UID: "d9a83dfc-0078-473a-b9bd-de9efc9a9633"). InnerVolumeSpecName "kube-api-access-5x5rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:06:14.315499 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.315366    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-dshm" (OuterVolumeSpecName: "dshm") pod "d9a83dfc-0078-473a-b9bd-de9efc9a9633" (UID: "d9a83dfc-0078-473a-b9bd-de9efc9a9633"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:14.325957 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.325921    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9a83dfc-0078-473a-b9bd-de9efc9a9633" (UID: "d9a83dfc-0078-473a-b9bd-de9efc9a9633"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:14.344172 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.344138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9a83dfc-0078-473a-b9bd-de9efc9a9633" (UID: "d9a83dfc-0078-473a-b9bd-de9efc9a9633"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:14.413660 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.413624    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.413660 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.413655    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5rh\" (UniqueName: \"kubernetes.io/projected/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kube-api-access-5x5rh\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.413660 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.413664    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.413883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.413673    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a83dfc-0078-473a-b9bd-de9efc9a9633-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.413883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.413681    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.413883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.413689    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.413883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.413697    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a83dfc-0078-473a-b9bd-de9efc9a9633-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.413883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.413705    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a83dfc-0078-473a-b9bd-de9efc9a9633-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.622933 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.622892    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"]
Sep 16 20:06:14.624116 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:14.624093    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9767b895-xtxtt"]
Sep 16 20:06:15.682113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:15.682078    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" path="/var/lib/kubelet/pods/d9a83dfc-0078-473a-b9bd-de9efc9a9633/volumes"
Sep 16 20:06:20.672063 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:20.672024    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:20.672528 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:20.672139    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:06:30.672640 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:30.672599    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:40.671867 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:40.671821    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:43.760749 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:43.760723    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:06:43.849812 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:43.849730    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx7lg\" (UniqueName: \"kubernetes.io/projected/09df674c-3745-4af5-9c67-cefda26542b2-kube-api-access-zx7lg\") pod \"09df674c-3745-4af5-9c67-cefda26542b2\" (UID: \"09df674c-3745-4af5-9c67-cefda26542b2\") "
Sep 16 20:06:43.849812 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:43.849772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09df674c-3745-4af5-9c67-cefda26542b2-tls-certs\") pod \"09df674c-3745-4af5-9c67-cefda26542b2\" (UID: \"09df674c-3745-4af5-9c67-cefda26542b2\") "
Sep 16 20:06:43.851854 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:43.851825    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09df674c-3745-4af5-9c67-cefda26542b2-kube-api-access-zx7lg" (OuterVolumeSpecName: "kube-api-access-zx7lg") pod "09df674c-3745-4af5-9c67-cefda26542b2" (UID: "09df674c-3745-4af5-9c67-cefda26542b2"). InnerVolumeSpecName "kube-api-access-zx7lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:06:43.851854 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:43.851828    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09df674c-3745-4af5-9c67-cefda26542b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09df674c-3745-4af5-9c67-cefda26542b2" (UID: "09df674c-3745-4af5-9c67-cefda26542b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:06:43.950489 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:43.950454    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx7lg\" (UniqueName: \"kubernetes.io/projected/09df674c-3745-4af5-9c67-cefda26542b2-kube-api-access-zx7lg\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:43.950489 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:43.950488    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09df674c-3745-4af5-9c67-cefda26542b2-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:06:44.413779 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.413745    2561 generic.go:356] "Generic (PLEG): container finished" podID="09df674c-3745-4af5-9c67-cefda26542b2" containerID="fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48" exitCode=0
Sep 16 20:06:44.413945 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.413808    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"
Sep 16 20:06:44.413945 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.413811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" event={"ID":"09df674c-3745-4af5-9c67-cefda26542b2","Type":"ContainerDied","Data":"fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48"}
Sep 16 20:06:44.414018 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.413949    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h" event={"ID":"09df674c-3745-4af5-9c67-cefda26542b2","Type":"ContainerDied","Data":"a89b4f480178c2c573065767964ab9ac1c5acd7f65c8c3ab8dff752441fd1ab6"}
Sep 16 20:06:44.414018 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.413971    2561 scope.go:117] "RemoveContainer" containerID="fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48"
Sep 16 20:06:44.421981 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.421963    2561 scope.go:117] "RemoveContainer" containerID="fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48"
Sep 16 20:06:44.422266 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:06:44.422241    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48\": container with ID starting with fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48 not found: ID does not exist" containerID="fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48"
Sep 16 20:06:44.422347 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.422277    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48"} err="failed to get container status \"fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48\": rpc error: code = NotFound desc = could not find container \"fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48\": container with ID starting with fc23b289038edbf78a4f67cd0b499df2b00942860bfb450461ca30e9b1228b48 not found: ID does not exist"
Sep 16 20:06:44.437927 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.437899    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"]
Sep 16 20:06:44.440241 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:44.440218    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-hqh2h"]
Sep 16 20:06:45.682469 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:45.682435    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09df674c-3745-4af5-9c67-cefda26542b2" path="/var/lib/kubelet/pods/09df674c-3745-4af5-9c67-cefda26542b2/volumes"
Sep 16 20:06:49.963237 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963206    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"]
Sep 16 20:06:49.963607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963472    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="storage-initializer"
Sep 16 20:06:49.963607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963483    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="storage-initializer"
Sep 16 20:06:49.963607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963495    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main"
Sep 16 20:06:49.963607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963501    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main"
Sep 16 20:06:49.963607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963510    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main"
Sep 16 20:06:49.963607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963516    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main"
Sep 16 20:06:49.963607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963577    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="09df674c-3745-4af5-9c67-cefda26542b2" containerName="main"
Sep 16 20:06:49.963607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.963585    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a83dfc-0078-473a-b9bd-de9efc9a9633" containerName="main"
Sep 16 20:06:49.966559 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.966536    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:49.975678 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.975655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 20:06:49.976918 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.976896    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 20:06:49.977041 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.976938    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 20:06:49.977041 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:49.976961    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-2945m\""
Sep 16 20:06:50.012750 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.012720    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"]
Sep 16 20:06:50.095590 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.095560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-home\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.095590 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.095596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-dshm\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.095767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.095626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.095767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.095690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqvk\" (UniqueName: \"kubernetes.io/projected/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-kube-api-access-4bqvk\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.095767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.095752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.095865 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.095805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-model-cache\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.196553 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.196506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.196553 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.196558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqvk\" (UniqueName: \"kubernetes.io/projected/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-kube-api-access-4bqvk\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.196768 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.196596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.196768 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.196629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-model-cache\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.196768 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.196659    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-home\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.196898 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.196871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-dshm\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.197034 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.197011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-model-cache\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.197102 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.197039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.197102 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.197052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-home\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.198936 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.198908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-dshm\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.199163 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.199147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.222269 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.222201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqvk\" (UniqueName: \"kubernetes.io/projected/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-kube-api-access-4bqvk\") pod \"canary-v1-kserve-65d58ccc9-h5chl\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.276057 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.276024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:06:50.559827 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.559771    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"]
Sep 16 20:06:50.562535 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:06:50.562503    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6523a91f_0e90_42a2_bb1d_76df6fc4f46d.slice/crio-ad24d441e3c4f0b909bd5757ac504802cafb42519522bcdf29cfdbc1460684f0 WatchSource:0}: Error finding container ad24d441e3c4f0b909bd5757ac504802cafb42519522bcdf29cfdbc1460684f0: Status 404 returned error can't find the container with id ad24d441e3c4f0b909bd5757ac504802cafb42519522bcdf29cfdbc1460684f0
Sep 16 20:06:50.564392 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:50.564356    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:06:51.442447 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:51.442403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl" event={"ID":"6523a91f-0e90-42a2-bb1d-76df6fc4f46d","Type":"ContainerStarted","Data":"ad24d441e3c4f0b909bd5757ac504802cafb42519522bcdf29cfdbc1460684f0"}
Sep 16 20:06:52.448155 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:52.448116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl" event={"ID":"6523a91f-0e90-42a2-bb1d-76df6fc4f46d","Type":"ContainerStarted","Data":"30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1"}
Sep 16 20:06:52.479393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:52.479325    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl" podStartSLOduration=2.277791478 podStartE2EDuration="3.479307967s" totalImagesPullingTime="1.201516489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:06:52.477030725 +0000 UTC m=+1659.248901135" watchObservedRunningTime="2026-09-16 20:06:52.479307967 +0000 UTC m=+1659.251178379"
Sep 16 20:06:53.276772 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:06:53.276718    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:07:00.276492 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:00.276449    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:07:00.288642 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:00.288615    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:07:00.489828 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:00.489797    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:07:47.481303 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.481221    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"]
Sep 16 20:07:47.484314 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.484293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.487694 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.487669    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 20:07:47.488900 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.488876    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 20:07:47.489090 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.488901    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-4j5x2\""
Sep 16 20:07:47.489207 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.488921    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 20:07:47.504774 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.504751    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"]
Sep 16 20:07:47.655415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.655357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7zq9\" (UniqueName: \"kubernetes.io/projected/32466345-4f73-4d6c-807e-f3a95fa6cd14-kube-api-access-k7zq9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.655595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.655424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.655595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.655477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.655595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.655509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32466345-4f73-4d6c-807e-f3a95fa6cd14-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.655595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.655538    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.655595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.655565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.756533 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.756492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7zq9\" (UniqueName: \"kubernetes.io/projected/32466345-4f73-4d6c-807e-f3a95fa6cd14-kube-api-access-k7zq9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.756709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.756546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.756709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.756572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.756709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.756591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32466345-4f73-4d6c-807e-f3a95fa6cd14-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.756709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.756625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.756924 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.756752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.757075 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.757046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.757188 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.757101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.757238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.757199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.758837 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.758816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.759448 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.759428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32466345-4f73-4d6c-807e-f3a95fa6cd14-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.769883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.769858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7zq9\" (UniqueName: \"kubernetes.io/projected/32466345-4f73-4d6c-807e-f3a95fa6cd14-kube-api-access-k7zq9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:47.795606 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:47.795577    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:48.073336 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:48.073237    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"]
Sep 16 20:07:48.076156 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:07:48.076122    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32466345_4f73_4d6c_807e_f3a95fa6cd14.slice/crio-bd498e303c2ba29019c01628a017482efe728ad1ca26c75d68462a4d61cd0d10 WatchSource:0}: Error finding container bd498e303c2ba29019c01628a017482efe728ad1ca26c75d68462a4d61cd0d10: Status 404 returned error can't find the container with id bd498e303c2ba29019c01628a017482efe728ad1ca26c75d68462a4d61cd0d10
Sep 16 20:07:48.681692 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:48.681655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" event={"ID":"32466345-4f73-4d6c-807e-f3a95fa6cd14","Type":"ContainerStarted","Data":"37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b"}
Sep 16 20:07:48.682072 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:48.681713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" event={"ID":"32466345-4f73-4d6c-807e-f3a95fa6cd14","Type":"ContainerStarted","Data":"bd498e303c2ba29019c01628a017482efe728ad1ca26c75d68462a4d61cd0d10"}
Sep 16 20:07:48.719992 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:48.719935    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" podStartSLOduration=1.7199186910000002 podStartE2EDuration="1.719918691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:07:48.7191745 +0000 UTC m=+1715.491044912" watchObservedRunningTime="2026-09-16 20:07:48.719918691 +0000 UTC m=+1715.491789101"
Sep 16 20:07:48.795852 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:48.795814    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:57.796442 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:57.796403    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:57.808848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:57.808823    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:07:58.730662 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:07:58.730630    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:08:28.644882 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:28.644849    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"]
Sep 16 20:08:28.653218 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:08:28.653185    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:08:28.653218 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:08:28.653218 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:08:28.653218 ip-10-0-136-120 kubenswrapper[2561]:  > containerID="37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b" cmd=["/bin/sleep","15"]
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:28.653233    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" message="" err=<
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]:  >
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:08:28.653246    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" podUID="32466345-4f73-4d6c-807e-f3a95fa6cd14" containerName="main" containerID="cri-o://37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b"
Sep 16 20:08:28.653464 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:28.653278    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" podUID="32466345-4f73-4d6c-807e-f3a95fa6cd14" containerName="main" containerID="cri-o://37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b" gracePeriod=60
Sep 16 20:08:28.720069 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:28.720027    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" podUID="32466345-4f73-4d6c-807e-f3a95fa6cd14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 20:08:28.993456 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:28.993431    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:08:29.052190 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-model-cache\") pod \"32466345-4f73-4d6c-807e-f3a95fa6cd14\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") "
Sep 16 20:08:29.052190 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-home\") pod \"32466345-4f73-4d6c-807e-f3a95fa6cd14\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") "
Sep 16 20:08:29.052435 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052234    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-dshm\") pod \"32466345-4f73-4d6c-807e-f3a95fa6cd14\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") "
Sep 16 20:08:29.052435 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-tmp-dir\") pod \"32466345-4f73-4d6c-807e-f3a95fa6cd14\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") "
Sep 16 20:08:29.052435 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052272    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7zq9\" (UniqueName: \"kubernetes.io/projected/32466345-4f73-4d6c-807e-f3a95fa6cd14-kube-api-access-k7zq9\") pod \"32466345-4f73-4d6c-807e-f3a95fa6cd14\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") "
Sep 16 20:08:29.052435 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052298    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32466345-4f73-4d6c-807e-f3a95fa6cd14-tls-certs\") pod \"32466345-4f73-4d6c-807e-f3a95fa6cd14\" (UID: \"32466345-4f73-4d6c-807e-f3a95fa6cd14\") "
Sep 16 20:08:29.052657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052486    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-model-cache" (OuterVolumeSpecName: "model-cache") pod "32466345-4f73-4d6c-807e-f3a95fa6cd14" (UID: "32466345-4f73-4d6c-807e-f3a95fa6cd14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:08:29.052657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32466345-4f73-4d6c-807e-f3a95fa6cd14" (UID: "32466345-4f73-4d6c-807e-f3a95fa6cd14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:08:29.052657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.052600    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-home" (OuterVolumeSpecName: "home") pod "32466345-4f73-4d6c-807e-f3a95fa6cd14" (UID: "32466345-4f73-4d6c-807e-f3a95fa6cd14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:08:29.054453 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.054431    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-dshm" (OuterVolumeSpecName: "dshm") pod "32466345-4f73-4d6c-807e-f3a95fa6cd14" (UID: "32466345-4f73-4d6c-807e-f3a95fa6cd14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:08:29.054522 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.054496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32466345-4f73-4d6c-807e-f3a95fa6cd14-kube-api-access-k7zq9" (OuterVolumeSpecName: "kube-api-access-k7zq9") pod "32466345-4f73-4d6c-807e-f3a95fa6cd14" (UID: "32466345-4f73-4d6c-807e-f3a95fa6cd14"). InnerVolumeSpecName "kube-api-access-k7zq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:08:29.054614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.054596    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32466345-4f73-4d6c-807e-f3a95fa6cd14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32466345-4f73-4d6c-807e-f3a95fa6cd14" (UID: "32466345-4f73-4d6c-807e-f3a95fa6cd14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:08:29.153032 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.152993    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.153032 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.153024    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.153032 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.153035    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7zq9\" (UniqueName: \"kubernetes.io/projected/32466345-4f73-4d6c-807e-f3a95fa6cd14-kube-api-access-k7zq9\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.153308 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.153044    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32466345-4f73-4d6c-807e-f3a95fa6cd14-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.153308 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.153052    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.153308 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.153060    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32466345-4f73-4d6c-807e-f3a95fa6cd14-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.855616 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.855579    2561 generic.go:356] "Generic (PLEG): container finished" podID="32466345-4f73-4d6c-807e-f3a95fa6cd14" containerID="37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b" exitCode=0
Sep 16 20:08:29.856034 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.855626    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"
Sep 16 20:08:29.856034 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.855642    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" event={"ID":"32466345-4f73-4d6c-807e-f3a95fa6cd14","Type":"ContainerDied","Data":"37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b"}
Sep 16 20:08:29.856034 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.855687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm" event={"ID":"32466345-4f73-4d6c-807e-f3a95fa6cd14","Type":"ContainerDied","Data":"bd498e303c2ba29019c01628a017482efe728ad1ca26c75d68462a4d61cd0d10"}
Sep 16 20:08:29.856034 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.855708    2561 scope.go:117] "RemoveContainer" containerID="37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b"
Sep 16 20:08:29.863683 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.863662    2561 scope.go:117] "RemoveContainer" containerID="37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b"
Sep 16 20:08:29.863974 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:08:29.863953    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b\": container with ID starting with 37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b not found: ID does not exist" containerID="37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b"
Sep 16 20:08:29.864043 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.863981    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b"} err="failed to get container status \"37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b\": rpc error: code = NotFound desc = could not find container \"37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b\": container with ID starting with 37b2d77ffe20273beb89b51797a3a83045a595b7d7a89bc5bf0639b161d1f85b not found: ID does not exist"
Sep 16 20:08:29.888261 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.888236    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"]
Sep 16 20:08:29.892344 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:29.892320    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gllbm"]
Sep 16 20:08:31.682096 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:08:31.682060    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32466345-4f73-4d6c-807e-f3a95fa6cd14" path="/var/lib/kubelet/pods/32466345-4f73-4d6c-807e-f3a95fa6cd14/volumes"
Sep 16 20:09:13.725912 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:13.725879    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:09:13.728284 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:13.728264    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:09:25.804033 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:25.803999    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"]
Sep 16 20:09:38.704814 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.704776    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"]
Sep 16 20:09:38.705253 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.705053    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32466345-4f73-4d6c-807e-f3a95fa6cd14" containerName="main"
Sep 16 20:09:38.705253 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.705064    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="32466345-4f73-4d6c-807e-f3a95fa6cd14" containerName="main"
Sep 16 20:09:38.705253 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.705139    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="32466345-4f73-4d6c-807e-f3a95fa6cd14" containerName="main"
Sep 16 20:09:38.707683 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.707662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.710322 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.710301    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:09:38.711482 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.711457    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 20:09:38.711482 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.711469    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-r7qzz\""
Sep 16 20:09:38.711627 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.711473    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 20:09:38.711627 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.711520    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 20:09:38.711627 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.711536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 20:09:38.721870 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.721847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"]
Sep 16 20:09:38.800328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.800287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tls-certs\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.800328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.800324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.800614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.800344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2147055-fbef-4ebf-bd26-7a88a7bbec81-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.800614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.800448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlx6s\" (UniqueName: \"kubernetes.io/projected/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kube-api-access-rlx6s\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.800614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.800492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-model-cache\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.800614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.800513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-home\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.800614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.800535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-dshm\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.800614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.800589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.901682 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.901642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tls-certs\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.901682 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.901682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.901903 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.901703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2147055-fbef-4ebf-bd26-7a88a7bbec81-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.901903 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.901731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx6s\" (UniqueName: \"kubernetes.io/projected/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kube-api-access-rlx6s\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.901903 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.901758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-model-cache\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.901903 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.901784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-home\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.901903 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.901808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-dshm\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.901903 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.901833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.902193 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.902156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.902193 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.902186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-home\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.902359 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.902248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.902432 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.902397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-model-cache\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.902568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.902549    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2147055-fbef-4ebf-bd26-7a88a7bbec81-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.904104 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.904084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-dshm\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.904275 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.904257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tls-certs\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.910504 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.910470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlx6s\" (UniqueName: \"kubernetes.io/projected/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kube-api-access-rlx6s\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-djm7c\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:38.972478 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.972390    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"]
Sep 16 20:09:38.976386 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.976349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:38.986359 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:38.986332    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"]
Sep 16 20:09:39.002217 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.002182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.002217 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.002226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d489415-05a2-44d3-bc66-f4b730311d64-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.002430 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.002276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d489415-05a2-44d3-bc66-f4b730311d64-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.002430 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.002332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.018392 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.018356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:39.103516 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.103481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d489415-05a2-44d3-bc66-f4b730311d64-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.103669 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.103546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d489415-05a2-44d3-bc66-f4b730311d64-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.103669 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.103572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.103669 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.103615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.104073 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.104030    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.104177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.104152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.104215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.104186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d489415-05a2-44d3-bc66-f4b730311d64-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.105869 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.105854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d489415-05a2-44d3-bc66-f4b730311d64-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.287806 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.287774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:09:39.293791 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.293676    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"]
Sep 16 20:09:39.297453 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:09:39.297415    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2147055_fbef_4ebf_bd26_7a88a7bbec81.slice/crio-ba2d969a47075d988345d1cd2523e4878aed701b562c16b519dd4c4dc9e656ce WatchSource:0}: Error finding container ba2d969a47075d988345d1cd2523e4878aed701b562c16b519dd4c4dc9e656ce: Status 404 returned error can't find the container with id ba2d969a47075d988345d1cd2523e4878aed701b562c16b519dd4c4dc9e656ce
Sep 16 20:09:39.561868 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:39.561758    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"]
Sep 16 20:09:39.564418 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:09:39.564393    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d489415_05a2_44d3_bc66_f4b730311d64.slice/crio-113da6bebd0250c5daabaf6a6ea595b0d65494e38c266f1c062b7388498e0307 WatchSource:0}: Error finding container 113da6bebd0250c5daabaf6a6ea595b0d65494e38c266f1c062b7388498e0307: Status 404 returned error can't find the container with id 113da6bebd0250c5daabaf6a6ea595b0d65494e38c266f1c062b7388498e0307
Sep 16 20:09:40.138126 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:40.138087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b" event={"ID":"5d489415-05a2-44d3-bc66-f4b730311d64","Type":"ContainerStarted","Data":"09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf"}
Sep 16 20:09:40.138626 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:40.138150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b" event={"ID":"5d489415-05a2-44d3-bc66-f4b730311d64","Type":"ContainerStarted","Data":"113da6bebd0250c5daabaf6a6ea595b0d65494e38c266f1c062b7388498e0307"}
Sep 16 20:09:40.139858 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:40.139833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" event={"ID":"f2147055-fbef-4ebf-bd26-7a88a7bbec81","Type":"ContainerStarted","Data":"69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512"}
Sep 16 20:09:40.139996 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:40.139879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" event={"ID":"f2147055-fbef-4ebf-bd26-7a88a7bbec81","Type":"ContainerStarted","Data":"ba2d969a47075d988345d1cd2523e4878aed701b562c16b519dd4c4dc9e656ce"}
Sep 16 20:09:40.827089 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:40.826899    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl" podUID="6523a91f-0e90-42a2-bb1d-76df6fc4f46d" containerName="main" containerID="cri-o://30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1" gracePeriod=45
Sep 16 20:09:41.081795 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.081735    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.124728    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-dshm\") pod \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") "
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.124780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-home\") pod \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") "
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.124862    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-model-cache\") pod \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") "
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.124905    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bqvk\" (UniqueName: \"kubernetes.io/projected/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-kube-api-access-4bqvk\") pod \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") "
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.124978    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tmp-dir\") pod \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") "
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.125010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tls-certs\") pod \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\" (UID: \"6523a91f-0e90-42a2-bb1d-76df6fc4f46d\") "
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.125713    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6523a91f-0e90-42a2-bb1d-76df6fc4f46d" (UID: "6523a91f-0e90-42a2-bb1d-76df6fc4f46d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.126358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-home" (OuterVolumeSpecName: "home") pod "6523a91f-0e90-42a2-bb1d-76df6fc4f46d" (UID: "6523a91f-0e90-42a2-bb1d-76df6fc4f46d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:41.132421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.126522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6523a91f-0e90-42a2-bb1d-76df6fc4f46d" (UID: "6523a91f-0e90-42a2-bb1d-76df6fc4f46d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:41.136854 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.134137    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-dshm" (OuterVolumeSpecName: "dshm") pod "6523a91f-0e90-42a2-bb1d-76df6fc4f46d" (UID: "6523a91f-0e90-42a2-bb1d-76df6fc4f46d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:41.136854 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.135832    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-kube-api-access-4bqvk" (OuterVolumeSpecName: "kube-api-access-4bqvk") pod "6523a91f-0e90-42a2-bb1d-76df6fc4f46d" (UID: "6523a91f-0e90-42a2-bb1d-76df6fc4f46d"). InnerVolumeSpecName "kube-api-access-4bqvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:09:41.138260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.137484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6523a91f-0e90-42a2-bb1d-76df6fc4f46d" (UID: "6523a91f-0e90-42a2-bb1d-76df6fc4f46d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:09:41.151230 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.148062    2561 generic.go:356] "Generic (PLEG): container finished" podID="6523a91f-0e90-42a2-bb1d-76df6fc4f46d" containerID="30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1" exitCode=0
Sep 16 20:09:41.151230 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.148873    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"
Sep 16 20:09:41.151230 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.149132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl" event={"ID":"6523a91f-0e90-42a2-bb1d-76df6fc4f46d","Type":"ContainerDied","Data":"30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1"}
Sep 16 20:09:41.151230 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.149184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl" event={"ID":"6523a91f-0e90-42a2-bb1d-76df6fc4f46d","Type":"ContainerDied","Data":"ad24d441e3c4f0b909bd5757ac504802cafb42519522bcdf29cfdbc1460684f0"}
Sep 16 20:09:41.151230 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.149214    2561 scope.go:117] "RemoveContainer" containerID="30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1"
Sep 16 20:09:41.168179 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.164667    2561 scope.go:117] "RemoveContainer" containerID="30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1"
Sep 16 20:09:41.170032 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:09:41.168505    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1\": container with ID starting with 30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1 not found: ID does not exist" containerID="30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1"
Sep 16 20:09:41.170032 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.168557    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1"} err="failed to get container status \"30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1\": rpc error: code = NotFound desc = could not find container \"30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1\": container with ID starting with 30ab4f2a5316ea3a54068a8753e76588187734097347fd3c91f5c5d7f0108ae1 not found: ID does not exist"
Sep 16 20:09:41.191945 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.191885    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"]
Sep 16 20:09:41.193633 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.193607    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-h5chl"]
Sep 16 20:09:41.227153 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.225987    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.227153 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.226018    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqvk\" (UniqueName: \"kubernetes.io/projected/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-kube-api-access-4bqvk\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.227153 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.226034    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.227153 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.226050    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.227153 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.226064    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.227153 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.226076    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6523a91f-0e90-42a2-bb1d-76df6fc4f46d-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.687858 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:41.687814    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6523a91f-0e90-42a2-bb1d-76df6fc4f46d" path="/var/lib/kubelet/pods/6523a91f-0e90-42a2-bb1d-76df6fc4f46d/volumes"
Sep 16 20:09:42.153309 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:42.153277    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d489415-05a2-44d3-bc66-f4b730311d64" containerID="09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf" exitCode=0
Sep 16 20:09:42.153707 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:42.153352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b" event={"ID":"5d489415-05a2-44d3-bc66-f4b730311d64","Type":"ContainerDied","Data":"09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf"}
Sep 16 20:09:42.154916 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:42.154896    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerID="69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512" exitCode=0
Sep 16 20:09:42.155016 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:42.154972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" event={"ID":"f2147055-fbef-4ebf-bd26-7a88a7bbec81","Type":"ContainerDied","Data":"69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512"}
Sep 16 20:09:43.165817 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:43.165781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" event={"ID":"f2147055-fbef-4ebf-bd26-7a88a7bbec81","Type":"ContainerStarted","Data":"82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7"}
Sep 16 20:09:43.190854 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:43.190757    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" podStartSLOduration=2.672386195 podStartE2EDuration="5.190737492s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.518351297s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:09:43.189168145 +0000 UTC m=+1829.961038558" watchObservedRunningTime="2026-09-16 20:09:43.190737492 +0000 UTC m=+1829.962607903"
Sep 16 20:09:44.019108 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:44.019061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:49.019252 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:49.019212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:49.037506 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:49.037452    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:09:49.216567 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:09:49.216532    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:10:29.399357 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:10:29.399316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b" event={"ID":"5d489415-05a2-44d3-bc66-f4b730311d64","Type":"ContainerStarted","Data":"4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb"}
Sep 16 20:10:29.399780 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:10:29.399579    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:10:29.420318 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:10:29.420270    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b" podStartSLOduration=2.746633287 podStartE2EDuration="51.420256482s" totalImagesPullingTime="46.176970135s" totalInitContainerRuntime="2.49665306s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:10:29.419454277 +0000 UTC m=+1876.191324685" watchObservedRunningTime="2026-09-16 20:10:29.420256482 +0000 UTC m=+1876.192126892"
Sep 16 20:11:00.414612 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:00.414576    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:11:04.886097 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:04.886063    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b_5d489415-05a2-44d3-bc66-f4b730311d64/main/0.log"
Sep 16 20:11:05.508427 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.508391    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"]
Sep 16 20:11:05.510329 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.510298    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"]
Sep 16 20:11:05.510576 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.510550    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b" podUID="5d489415-05a2-44d3-bc66-f4b730311d64" containerName="main" containerID="cri-o://4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb" gracePeriod=30
Sep 16 20:11:05.517062 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:11:05.517023    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:11:05.517062 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:11:05.517062 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:11:05.517062 ip-10-0-136-120 kubenswrapper[2561]:  > containerID="82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7" cmd=["/bin/sleep","15"]
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.517094    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" message="" err=<
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]:  >
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:11:05.517116    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" podUID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerName="main" containerID="cri-o://82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7"
Sep 16 20:11:05.517326 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.517162    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" podUID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerName="main" containerID="cri-o://82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7" gracePeriod=60
Sep 16 20:11:05.864567 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.864530    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:11:05.960164 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960136    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlx6s\" (UniqueName: \"kubernetes.io/projected/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kube-api-access-rlx6s\") pod \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") "
Sep 16 20:11:05.960564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960170    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-home\") pod \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") "
Sep 16 20:11:05.960564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tmp-dir\") pod \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") "
Sep 16 20:11:05.960564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-model-cache\") pod \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") "
Sep 16 20:11:05.960564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960260    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2147055-fbef-4ebf-bd26-7a88a7bbec81-cabundle-cert\") pod \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") "
Sep 16 20:11:05.960564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960298    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kserve-provision-location\") pod \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") "
Sep 16 20:11:05.960564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960343    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-dshm\") pod \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") "
Sep 16 20:11:05.960564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960401    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tls-certs\") pod \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\" (UID: \"f2147055-fbef-4ebf-bd26-7a88a7bbec81\") "
Sep 16 20:11:05.960564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960456    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-home" (OuterVolumeSpecName: "home") pod "f2147055-fbef-4ebf-bd26-7a88a7bbec81" (UID: "f2147055-fbef-4ebf-bd26-7a88a7bbec81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:05.960906 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960577    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2147055-fbef-4ebf-bd26-7a88a7bbec81" (UID: "f2147055-fbef-4ebf-bd26-7a88a7bbec81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:05.960906 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960655    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:05.960906 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.960674    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2147055-fbef-4ebf-bd26-7a88a7bbec81" (UID: "f2147055-fbef-4ebf-bd26-7a88a7bbec81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:05.961095 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.961069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2147055-fbef-4ebf-bd26-7a88a7bbec81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2147055-fbef-4ebf-bd26-7a88a7bbec81" (UID: "f2147055-fbef-4ebf-bd26-7a88a7bbec81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:11:05.962711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.962677    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kube-api-access-rlx6s" (OuterVolumeSpecName: "kube-api-access-rlx6s") pod "f2147055-fbef-4ebf-bd26-7a88a7bbec81" (UID: "f2147055-fbef-4ebf-bd26-7a88a7bbec81"). InnerVolumeSpecName "kube-api-access-rlx6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:11:05.962816 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.962784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-dshm" (OuterVolumeSpecName: "dshm") pod "f2147055-fbef-4ebf-bd26-7a88a7bbec81" (UID: "f2147055-fbef-4ebf-bd26-7a88a7bbec81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:05.962911 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.962889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2147055-fbef-4ebf-bd26-7a88a7bbec81" (UID: "f2147055-fbef-4ebf-bd26-7a88a7bbec81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:11:05.992557 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:05.992495    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2147055-fbef-4ebf-bd26-7a88a7bbec81" (UID: "f2147055-fbef-4ebf-bd26-7a88a7bbec81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:06.061728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.061657    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.061728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.061683    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2147055-fbef-4ebf-bd26-7a88a7bbec81-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.061728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.061693    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.061728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.061703    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.061728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.061711    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.061728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.061720    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlx6s\" (UniqueName: \"kubernetes.io/projected/f2147055-fbef-4ebf-bd26-7a88a7bbec81-kube-api-access-rlx6s\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.061728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.061727    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2147055-fbef-4ebf-bd26-7a88a7bbec81-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.079642 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.079619    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:11:06.262955 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.262921    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-tmp-dir\") pod \"5d489415-05a2-44d3-bc66-f4b730311d64\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") "
Sep 16 20:11:06.263131 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.262963    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-kserve-provision-location\") pod \"5d489415-05a2-44d3-bc66-f4b730311d64\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") "
Sep 16 20:11:06.263131 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.262986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d489415-05a2-44d3-bc66-f4b730311d64-cabundle-cert\") pod \"5d489415-05a2-44d3-bc66-f4b730311d64\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") "
Sep 16 20:11:06.263131 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.263008    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d489415-05a2-44d3-bc66-f4b730311d64-tls-certs\") pod \"5d489415-05a2-44d3-bc66-f4b730311d64\" (UID: \"5d489415-05a2-44d3-bc66-f4b730311d64\") "
Sep 16 20:11:06.263348 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.263320    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d489415-05a2-44d3-bc66-f4b730311d64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d489415-05a2-44d3-bc66-f4b730311d64" (UID: "5d489415-05a2-44d3-bc66-f4b730311d64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:11:06.263442 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.263407    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d489415-05a2-44d3-bc66-f4b730311d64" (UID: "5d489415-05a2-44d3-bc66-f4b730311d64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:06.265064 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.265043    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d489415-05a2-44d3-bc66-f4b730311d64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d489415-05a2-44d3-bc66-f4b730311d64" (UID: "5d489415-05a2-44d3-bc66-f4b730311d64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:11:06.283751 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.283717    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d489415-05a2-44d3-bc66-f4b730311d64" (UID: "5d489415-05a2-44d3-bc66-f4b730311d64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:06.364393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.364292    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.364393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.364323    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d489415-05a2-44d3-bc66-f4b730311d64-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.364393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.364333    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d489415-05a2-44d3-bc66-f4b730311d64-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.364393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.364341    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d489415-05a2-44d3-bc66-f4b730311d64-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.558599 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.558527    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d489415-05a2-44d3-bc66-f4b730311d64" containerID="4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb" exitCode=0
Sep 16 20:11:06.558599 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.558586    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"
Sep 16 20:11:06.558842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.558609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b" event={"ID":"5d489415-05a2-44d3-bc66-f4b730311d64","Type":"ContainerDied","Data":"4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb"}
Sep 16 20:11:06.558842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.558695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b" event={"ID":"5d489415-05a2-44d3-bc66-f4b730311d64","Type":"ContainerDied","Data":"113da6bebd0250c5daabaf6a6ea595b0d65494e38c266f1c062b7388498e0307"}
Sep 16 20:11:06.558842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.558736    2561 scope.go:117] "RemoveContainer" containerID="4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb"
Sep 16 20:11:06.560767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.560740    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerID="82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7" exitCode=0
Sep 16 20:11:06.560889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.560786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" event={"ID":"f2147055-fbef-4ebf-bd26-7a88a7bbec81","Type":"ContainerDied","Data":"82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7"}
Sep 16 20:11:06.560889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.560798    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"
Sep 16 20:11:06.560889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.560824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c" event={"ID":"f2147055-fbef-4ebf-bd26-7a88a7bbec81","Type":"ContainerDied","Data":"ba2d969a47075d988345d1cd2523e4878aed701b562c16b519dd4c4dc9e656ce"}
Sep 16 20:11:06.571285 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.568964    2561 scope.go:117] "RemoveContainer" containerID="09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf"
Sep 16 20:11:06.580349 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.580330    2561 scope.go:117] "RemoveContainer" containerID="4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb"
Sep 16 20:11:06.580655 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:11:06.580636    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb\": container with ID starting with 4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb not found: ID does not exist" containerID="4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb"
Sep 16 20:11:06.580708 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.580663    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb"} err="failed to get container status \"4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb\": rpc error: code = NotFound desc = could not find container \"4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb\": container with ID starting with 4aef00b37897eab8268d553f021ee1da14626ca5567a6fb13798ae1c861f5bbb not found: ID does not exist"
Sep 16 20:11:06.580708 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.580680    2561 scope.go:117] "RemoveContainer" containerID="09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf"
Sep 16 20:11:06.580947 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:11:06.580929    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf\": container with ID starting with 09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf not found: ID does not exist" containerID="09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf"
Sep 16 20:11:06.580994 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.580953    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf"} err="failed to get container status \"09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf\": rpc error: code = NotFound desc = could not find container \"09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf\": container with ID starting with 09ee06ebb39c5f490e349454b3e225893bd932c327a7bb9235ad4c638431e7cf not found: ID does not exist"
Sep 16 20:11:06.580994 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.580972    2561 scope.go:117] "RemoveContainer" containerID="82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7"
Sep 16 20:11:06.588339 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.588314    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"]
Sep 16 20:11:06.589104 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.589088    2561 scope.go:117] "RemoveContainer" containerID="69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512"
Sep 16 20:11:06.592703 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.592684    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c57c87f9b-n8x8b"]
Sep 16 20:11:06.596361 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.596331    2561 scope.go:117] "RemoveContainer" containerID="82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7"
Sep 16 20:11:06.596654 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:11:06.596636    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7\": container with ID starting with 82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7 not found: ID does not exist" containerID="82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7"
Sep 16 20:11:06.596707 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.596660    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7"} err="failed to get container status \"82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7\": rpc error: code = NotFound desc = could not find container \"82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7\": container with ID starting with 82256fc794eae214e6b3b338e410f47b34a3e407601d9b232dd68b8f979ec5d7 not found: ID does not exist"
Sep 16 20:11:06.596707 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.596676    2561 scope.go:117] "RemoveContainer" containerID="69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512"
Sep 16 20:11:06.596914 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:11:06.596895    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512\": container with ID starting with 69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512 not found: ID does not exist" containerID="69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512"
Sep 16 20:11:06.596971 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.596922    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512"} err="failed to get container status \"69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512\": rpc error: code = NotFound desc = could not find container \"69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512\": container with ID starting with 69721559503bcc538defe6516ba8221c49cc7114a72b029e10650aebf5e0b512 not found: ID does not exist"
Sep 16 20:11:06.609750 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.609722    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"]
Sep 16 20:11:06.631191 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:06.631170    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-djm7c"]
Sep 16 20:11:07.682692 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:07.682662    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d489415-05a2-44d3-bc66-f4b730311d64" path="/var/lib/kubelet/pods/5d489415-05a2-44d3-bc66-f4b730311d64/volumes"
Sep 16 20:11:07.683111 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:07.683098    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" path="/var/lib/kubelet/pods/f2147055-fbef-4ebf-bd26-7a88a7bbec81/volumes"
Sep 16 20:11:20.966855 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.966814    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"]
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967101    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6523a91f-0e90-42a2-bb1d-76df6fc4f46d" containerName="main"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967114    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6523a91f-0e90-42a2-bb1d-76df6fc4f46d" containerName="main"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967122    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d489415-05a2-44d3-bc66-f4b730311d64" containerName="storage-initializer"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967128    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d489415-05a2-44d3-bc66-f4b730311d64" containerName="storage-initializer"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967138    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerName="main"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967147    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerName="main"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967159    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerName="storage-initializer"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967164    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerName="storage-initializer"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967171    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d489415-05a2-44d3-bc66-f4b730311d64" containerName="main"
Sep 16 20:11:20.967215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967176    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d489415-05a2-44d3-bc66-f4b730311d64" containerName="main"
Sep 16 20:11:20.967601 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967236    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6523a91f-0e90-42a2-bb1d-76df6fc4f46d" containerName="main"
Sep 16 20:11:20.967601 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967244    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2147055-fbef-4ebf-bd26-7a88a7bbec81" containerName="main"
Sep 16 20:11:20.967601 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.967250    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d489415-05a2-44d3-bc66-f4b730311d64" containerName="main"
Sep 16 20:11:20.968927 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.968911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:20.973199 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.973177    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 20:11:20.973346 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.973199    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 20:11:20.973346 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.973278    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-zlvf9\""
Sep 16 20:11:20.973488 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.973471    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 20:11:20.992499 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:20.992473    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"]
Sep 16 20:11:21.060393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.060341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-home\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.060568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.060405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705e7f11-204e-4609-9ecd-3526ba9840dd-tls-certs\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.060568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.060430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-model-cache\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.060568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.060455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46z9q\" (UniqueName: \"kubernetes.io/projected/705e7f11-204e-4609-9ecd-3526ba9840dd-kube-api-access-46z9q\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.060568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.060504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-dshm\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.060568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.060520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161212 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705e7f11-204e-4609-9ecd-3526ba9840dd-tls-certs\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161212 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-model-cache\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46z9q\" (UniqueName: \"kubernetes.io/projected/705e7f11-204e-4609-9ecd-3526ba9840dd-kube-api-access-46z9q\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-dshm\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-home\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161748 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-model-cache\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161748 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-home\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.161825 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.161780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.163562 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.163540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-dshm\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.163923 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.163901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705e7f11-204e-4609-9ecd-3526ba9840dd-tls-certs\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.170750 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.170725    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46z9q\" (UniqueName: \"kubernetes.io/projected/705e7f11-204e-4609-9ecd-3526ba9840dd-kube-api-access-46z9q\") pod \"test-div-v1-kserve-fb566d667-pzpdm\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.280423 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.280391    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:21.561474 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.561448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"]
Sep 16 20:11:21.563583 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:11:21.563556    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705e7f11_204e_4609_9ecd_3526ba9840dd.slice/crio-0b72bf2d0df10cb2409196da0886cceb5b6696d4bd7380fada240c059e52de03 WatchSource:0}: Error finding container 0b72bf2d0df10cb2409196da0886cceb5b6696d4bd7380fada240c059e52de03: Status 404 returned error can't find the container with id 0b72bf2d0df10cb2409196da0886cceb5b6696d4bd7380fada240c059e52de03
Sep 16 20:11:21.624074 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.624041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm" event={"ID":"705e7f11-204e-4609-9ecd-3526ba9840dd","Type":"ContainerStarted","Data":"100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c"}
Sep 16 20:11:21.624173 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.624103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm" event={"ID":"705e7f11-204e-4609-9ecd-3526ba9840dd","Type":"ContainerStarted","Data":"0b72bf2d0df10cb2409196da0886cceb5b6696d4bd7380fada240c059e52de03"}
Sep 16 20:11:21.652667 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:21.652621    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm" podStartSLOduration=1.652605693 podStartE2EDuration="1.652605693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:11:21.652056388 +0000 UTC m=+1928.423926812" watchObservedRunningTime="2026-09-16 20:11:21.652605693 +0000 UTC m=+1928.424476103"
Sep 16 20:11:22.280885 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:22.280846    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:31.281156 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:31.281115    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:31.293349 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:31.293327    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:31.676724 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:31.676639    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:11:51.421738 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:11:51.421698    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"]
Sep 16 20:12:06.440937 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.440847    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm" podUID="705e7f11-204e-4609-9ecd-3526ba9840dd" containerName="main" containerID="cri-o://100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c" gracePeriod=45
Sep 16 20:12:06.678231 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.678205    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:12:06.804283 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.804247    2561 generic.go:356] "Generic (PLEG): container finished" podID="705e7f11-204e-4609-9ecd-3526ba9840dd" containerID="100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c" exitCode=0
Sep 16 20:12:06.804501 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.804291    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"
Sep 16 20:12:06.804501 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.804290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm" event={"ID":"705e7f11-204e-4609-9ecd-3526ba9840dd","Type":"ContainerDied","Data":"100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c"}
Sep 16 20:12:06.804501 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.804437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm" event={"ID":"705e7f11-204e-4609-9ecd-3526ba9840dd","Type":"ContainerDied","Data":"0b72bf2d0df10cb2409196da0886cceb5b6696d4bd7380fada240c059e52de03"}
Sep 16 20:12:06.804501 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.804468    2561 scope.go:117] "RemoveContainer" containerID="100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c"
Sep 16 20:12:06.807726 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.807710    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-tmp-dir\") pod \"705e7f11-204e-4609-9ecd-3526ba9840dd\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") "
Sep 16 20:12:06.807821 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.807750    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-home\") pod \"705e7f11-204e-4609-9ecd-3526ba9840dd\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") "
Sep 16 20:12:06.807821 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.807783    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-model-cache\") pod \"705e7f11-204e-4609-9ecd-3526ba9840dd\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") "
Sep 16 20:12:06.807821 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.807802    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46z9q\" (UniqueName: \"kubernetes.io/projected/705e7f11-204e-4609-9ecd-3526ba9840dd-kube-api-access-46z9q\") pod \"705e7f11-204e-4609-9ecd-3526ba9840dd\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") "
Sep 16 20:12:06.807941 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.807832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705e7f11-204e-4609-9ecd-3526ba9840dd-tls-certs\") pod \"705e7f11-204e-4609-9ecd-3526ba9840dd\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") "
Sep 16 20:12:06.807941 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.807862    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-dshm\") pod \"705e7f11-204e-4609-9ecd-3526ba9840dd\" (UID: \"705e7f11-204e-4609-9ecd-3526ba9840dd\") "
Sep 16 20:12:06.808083 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.808042    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "705e7f11-204e-4609-9ecd-3526ba9840dd" (UID: "705e7f11-204e-4609-9ecd-3526ba9840dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:06.808083 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.808050    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-home" (OuterVolumeSpecName: "home") pod "705e7f11-204e-4609-9ecd-3526ba9840dd" (UID: "705e7f11-204e-4609-9ecd-3526ba9840dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:06.808268 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.808072    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "705e7f11-204e-4609-9ecd-3526ba9840dd" (UID: "705e7f11-204e-4609-9ecd-3526ba9840dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:06.808268 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.808247    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:12:06.808268 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.808265    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:12:06.808455 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.808276    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:12:06.810570 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.810533    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-dshm" (OuterVolumeSpecName: "dshm") pod "705e7f11-204e-4609-9ecd-3526ba9840dd" (UID: "705e7f11-204e-4609-9ecd-3526ba9840dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:06.810676 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.810636    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705e7f11-204e-4609-9ecd-3526ba9840dd-kube-api-access-46z9q" (OuterVolumeSpecName: "kube-api-access-46z9q") pod "705e7f11-204e-4609-9ecd-3526ba9840dd" (UID: "705e7f11-204e-4609-9ecd-3526ba9840dd"). InnerVolumeSpecName "kube-api-access-46z9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:12:06.810738 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.810715    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705e7f11-204e-4609-9ecd-3526ba9840dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "705e7f11-204e-4609-9ecd-3526ba9840dd" (UID: "705e7f11-204e-4609-9ecd-3526ba9840dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:12:06.813242 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.813224    2561 scope.go:117] "RemoveContainer" containerID="100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c"
Sep 16 20:12:06.813576 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:12:06.813557    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c\": container with ID starting with 100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c not found: ID does not exist" containerID="100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c"
Sep 16 20:12:06.813652 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.813584    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c"} err="failed to get container status \"100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c\": rpc error: code = NotFound desc = could not find container \"100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c\": container with ID starting with 100fa221bcb35d0cd59bc032c45edbd4ce70278229e4fb09087481f5c39b8d6c not found: ID does not exist"
Sep 16 20:12:06.909189 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.909154    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705e7f11-204e-4609-9ecd-3526ba9840dd-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:12:06.909189 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.909182    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705e7f11-204e-4609-9ecd-3526ba9840dd-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:12:06.909189 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:06.909193    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46z9q\" (UniqueName: \"kubernetes.io/projected/705e7f11-204e-4609-9ecd-3526ba9840dd-kube-api-access-46z9q\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:12:07.128570 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:07.128536    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"]
Sep 16 20:12:07.135220 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:07.135193    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm"]
Sep 16 20:12:07.666915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:07.666871    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-pzpdm" podUID="705e7f11-204e-4609-9ecd-3526ba9840dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": context deadline exceeded"
Sep 16 20:12:07.682811 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:07.682781    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="705e7f11-204e-4609-9ecd-3526ba9840dd" path="/var/lib/kubelet/pods/705e7f11-204e-4609-9ecd-3526ba9840dd/volumes"
Sep 16 20:12:22.501314 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.501276    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"]
Sep 16 20:12:22.501701 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.501671    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="705e7f11-204e-4609-9ecd-3526ba9840dd" containerName="main"
Sep 16 20:12:22.501701 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.501686    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="705e7f11-204e-4609-9ecd-3526ba9840dd" containerName="main"
Sep 16 20:12:22.501770 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.501763    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="705e7f11-204e-4609-9ecd-3526ba9840dd" containerName="main"
Sep 16 20:12:22.504734 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.504716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.508673 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.508647    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 16 20:12:22.508819 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.508772    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 20:12:22.509806 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.509786    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-rxfcp\""
Sep 16 20:12:22.509906 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.509864    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 20:12:22.519151 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.519127    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"]
Sep 16 20:12:22.632039 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.632006    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-home\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.632225 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.632054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hzf\" (UniqueName: \"kubernetes.io/projected/d0677295-7e57-4812-91a6-962b5d4a7e9f-kube-api-access-27hzf\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.632225 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.632119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.632225 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.632188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-model-cache\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.632354 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.632235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-dshm\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.632354 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.632267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0677295-7e57-4812-91a6-962b5d4a7e9f-tls-certs\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.732611 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.732573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-dshm\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.732777 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.732626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0677295-7e57-4812-91a6-962b5d4a7e9f-tls-certs\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.732777 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.732662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-home\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.732777 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.732691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hzf\" (UniqueName: \"kubernetes.io/projected/d0677295-7e57-4812-91a6-962b5d4a7e9f-kube-api-access-27hzf\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.732777 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.732724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.732777 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.732756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-model-cache\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.733099 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.733077    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-home\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.733296 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.733276    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.733388 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.733299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-model-cache\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.735074 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.735049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-dshm\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.735240 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.735225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0677295-7e57-4812-91a6-962b5d4a7e9f-tls-certs\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.741139 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.741117    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hzf\" (UniqueName: \"kubernetes.io/projected/d0677295-7e57-4812-91a6-962b5d4a7e9f-kube-api-access-27hzf\") pod \"stop-v2-kserve-7f68747c45-ntm9n\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:22.815236 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:22.815150    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:23.090331 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:23.090306    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"]
Sep 16 20:12:23.092698 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:12:23.092664    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0677295_7e57_4812_91a6_962b5d4a7e9f.slice/crio-1fa53133597bcdc4d697d3ab7c7f2d2941902ea697daf75227eac3e612f79772 WatchSource:0}: Error finding container 1fa53133597bcdc4d697d3ab7c7f2d2941902ea697daf75227eac3e612f79772: Status 404 returned error can't find the container with id 1fa53133597bcdc4d697d3ab7c7f2d2941902ea697daf75227eac3e612f79772
Sep 16 20:12:23.875502 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:23.875462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n" event={"ID":"d0677295-7e57-4812-91a6-962b5d4a7e9f","Type":"ContainerStarted","Data":"532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719"}
Sep 16 20:12:23.875877 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:23.875516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n" event={"ID":"d0677295-7e57-4812-91a6-962b5d4a7e9f","Type":"ContainerStarted","Data":"1fa53133597bcdc4d697d3ab7c7f2d2941902ea697daf75227eac3e612f79772"}
Sep 16 20:12:23.894339 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:23.894283    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n" podStartSLOduration=1.8942689590000001 podStartE2EDuration="1.894268959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:12:23.893273831 +0000 UTC m=+1990.665144243" watchObservedRunningTime="2026-09-16 20:12:23.894268959 +0000 UTC m=+1990.666139414"
Sep 16 20:12:24.815933 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:24.815885    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:32.815623 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:32.815558    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:32.827974 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:32.827947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:32.929068 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:32.929030    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:12:34.690491 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.690455    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"]
Sep 16 20:12:34.694949 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.694928    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.697748 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.697728    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 20:12:34.699092 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.699068    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 20:12:34.699092 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.699082    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 20:12:34.699239 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.699158    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:12:34.699239 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.699214    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-xzlb2\""
Sep 16 20:12:34.699465 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.699440    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 20:12:34.709777 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.709754    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"]
Sep 16 20:12:34.830841 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.830805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.830841 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.830848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dm7n\" (UniqueName: \"kubernetes.io/projected/9ef4c9a2-e7df-46e3-8053-fde543232617-kube-api-access-7dm7n\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.831060 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.830880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.831060 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.830940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-model-cache\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.831060 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.830974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-dshm\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.831060 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.830995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ef4c9a2-e7df-46e3-8053-fde543232617-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.831060 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.831041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef4c9a2-e7df-46e3-8053-fde543232617-tls-certs\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.831226 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.831079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-home\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.931733 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.931701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.931733 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.931739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dm7n\" (UniqueName: \"kubernetes.io/projected/9ef4c9a2-e7df-46e3-8053-fde543232617-kube-api-access-7dm7n\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932026 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.931770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932026 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.931803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-model-cache\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932026 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.931822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-dshm\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932026 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.931843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ef4c9a2-e7df-46e3-8053-fde543232617-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932026 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.931870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef4c9a2-e7df-46e3-8053-fde543232617-tls-certs\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932026 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.931997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-home\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.932205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932413 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.932386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-model-cache\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932471 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.932452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-home\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932642 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.932621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ef4c9a2-e7df-46e3-8053-fde543232617-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.932950 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.932928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.934669 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.934645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef4c9a2-e7df-46e3-8053-fde543232617-tls-certs\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.934760 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.934685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-dshm\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.945401 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.945329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dm7n\" (UniqueName: \"kubernetes.io/projected/9ef4c9a2-e7df-46e3-8053-fde543232617-kube-api-access-7dm7n\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:34.984185 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.984147    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"]
Sep 16 20:12:34.988354 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:34.988337    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.005545 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.005521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:35.011199 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.010951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"]
Sep 16 20:12:35.134051 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.134011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f604a7c4-28e1-430c-adf1-cc2be30fa464-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.134051 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.134051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.134295 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.134088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f604a7c4-28e1-430c-adf1-cc2be30fa464-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.134295 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.134142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.234855 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.234754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f604a7c4-28e1-430c-adf1-cc2be30fa464-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.234855 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.234808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.235085 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.234878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f604a7c4-28e1-430c-adf1-cc2be30fa464-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.235085 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.234904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.235278 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.235252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.235345 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.235301    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.235550 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.235533    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f604a7c4-28e1-430c-adf1-cc2be30fa464-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.237235 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.237216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f604a7c4-28e1-430c-adf1-cc2be30fa464-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.288307 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.288278    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"]
Sep 16 20:12:35.290778 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:12:35.290755    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef4c9a2_e7df_46e3_8053_fde543232617.slice/crio-67f9a44213ffdf04ba9f784215bfebe1c7055d53b057158c313e9b2625c9f601 WatchSource:0}: Error finding container 67f9a44213ffdf04ba9f784215bfebe1c7055d53b057158c313e9b2625c9f601: Status 404 returned error can't find the container with id 67f9a44213ffdf04ba9f784215bfebe1c7055d53b057158c313e9b2625c9f601
Sep 16 20:12:35.300415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.300398    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:35.577676 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.577623    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"]
Sep 16 20:12:35.580257 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:12:35.580228    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf604a7c4_28e1_430c_adf1_cc2be30fa464.slice/crio-0805d2efdeda071945190c0d508e40b00dfad47573d89d2e69d31c84e3e7888b WatchSource:0}: Error finding container 0805d2efdeda071945190c0d508e40b00dfad47573d89d2e69d31c84e3e7888b: Status 404 returned error can't find the container with id 0805d2efdeda071945190c0d508e40b00dfad47573d89d2e69d31c84e3e7888b
Sep 16 20:12:35.934712 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.934611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6" event={"ID":"f604a7c4-28e1-430c-adf1-cc2be30fa464","Type":"ContainerStarted","Data":"8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87"}
Sep 16 20:12:35.934712 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.934692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6" event={"ID":"f604a7c4-28e1-430c-adf1-cc2be30fa464","Type":"ContainerStarted","Data":"0805d2efdeda071945190c0d508e40b00dfad47573d89d2e69d31c84e3e7888b"}
Sep 16 20:12:35.936495 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.936462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" event={"ID":"9ef4c9a2-e7df-46e3-8053-fde543232617","Type":"ContainerStarted","Data":"f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3"}
Sep 16 20:12:35.936659 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:35.936519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" event={"ID":"9ef4c9a2-e7df-46e3-8053-fde543232617","Type":"ContainerStarted","Data":"67f9a44213ffdf04ba9f784215bfebe1c7055d53b057158c313e9b2625c9f601"}
Sep 16 20:12:37.949127 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:37.949093    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerID="f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3" exitCode=0
Sep 16 20:12:37.949559 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:37.949179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" event={"ID":"9ef4c9a2-e7df-46e3-8053-fde543232617","Type":"ContainerDied","Data":"f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3"}
Sep 16 20:12:38.954693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:38.954656    2561 generic.go:356] "Generic (PLEG): container finished" podID="f604a7c4-28e1-430c-adf1-cc2be30fa464" containerID="8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87" exitCode=0
Sep 16 20:12:38.955201 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:38.954728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6" event={"ID":"f604a7c4-28e1-430c-adf1-cc2be30fa464","Type":"ContainerDied","Data":"8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87"}
Sep 16 20:12:38.956803 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:38.956780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" event={"ID":"9ef4c9a2-e7df-46e3-8053-fde543232617","Type":"ContainerStarted","Data":"acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1"}
Sep 16 20:12:38.995592 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:38.995542    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" podStartSLOduration=2.499077034 podStartE2EDuration="4.995526883s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.496449849s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:12:38.994112033 +0000 UTC m=+2005.765982445" watchObservedRunningTime="2026-09-16 20:12:38.995526883 +0000 UTC m=+2005.767397294"
Sep 16 20:12:39.006266 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:39.006241    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:39.964109 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:39.964073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6" event={"ID":"f604a7c4-28e1-430c-adf1-cc2be30fa464","Type":"ContainerStarted","Data":"fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2"}
Sep 16 20:12:39.964512 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:39.964437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:12:39.987873 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:39.987803    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6" podStartSLOduration=3.50798151 podStartE2EDuration="5.987786573s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.479805063s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:12:39.986218184 +0000 UTC m=+2006.758088596" watchObservedRunningTime="2026-09-16 20:12:39.987786573 +0000 UTC m=+2006.759656983"
Sep 16 20:12:45.006603 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:45.006565    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:45.019601 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:45.019574    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:12:46.002504 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:12:46.002473    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:13:10.981053 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:10.981019    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:13:38.332251 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:38.332160    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6_f604a7c4-28e1-430c-adf1-cc2be30fa464/main/0.log"
Sep 16 20:13:40.077979 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.077944    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"]
Sep 16 20:13:40.081321 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.081288    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"]
Sep 16 20:13:40.081561 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.081525    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6" podUID="f604a7c4-28e1-430c-adf1-cc2be30fa464" containerName="main" containerID="cri-o://fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2" gracePeriod=30
Sep 16 20:13:40.086735 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:13:40.086703    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:13:40.086735 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:13:40.086735 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:13:40.086735 ip-10-0-136-120 kubenswrapper[2561]:  > containerID="acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1" cmd=["/bin/sleep","15"]
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.086753    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" message="" err=<
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]:  >
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:13:40.086764    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" podUID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerName="main" containerID="cri-o://acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1"
Sep 16 20:13:40.086927 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.086800    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" podUID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerName="main" containerID="cri-o://acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1" gracePeriod=60
Sep 16 20:13:40.431530 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.431505    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:13:40.494163 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ef4c9a2-e7df-46e3-8053-fde543232617-cabundle-cert\") pod \"9ef4c9a2-e7df-46e3-8053-fde543232617\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") "
Sep 16 20:13:40.494343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494183    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-tmp-dir\") pod \"9ef4c9a2-e7df-46e3-8053-fde543232617\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") "
Sep 16 20:13:40.494343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dm7n\" (UniqueName: \"kubernetes.io/projected/9ef4c9a2-e7df-46e3-8053-fde543232617-kube-api-access-7dm7n\") pod \"9ef4c9a2-e7df-46e3-8053-fde543232617\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") "
Sep 16 20:13:40.494343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494256    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef4c9a2-e7df-46e3-8053-fde543232617-tls-certs\") pod \"9ef4c9a2-e7df-46e3-8053-fde543232617\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") "
Sep 16 20:13:40.494343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-kserve-provision-location\") pod \"9ef4c9a2-e7df-46e3-8053-fde543232617\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") "
Sep 16 20:13:40.494343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494312    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-model-cache\") pod \"9ef4c9a2-e7df-46e3-8053-fde543232617\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") "
Sep 16 20:13:40.494657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-dshm\") pod \"9ef4c9a2-e7df-46e3-8053-fde543232617\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") "
Sep 16 20:13:40.494657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-home\") pod \"9ef4c9a2-e7df-46e3-8053-fde543232617\" (UID: \"9ef4c9a2-e7df-46e3-8053-fde543232617\") "
Sep 16 20:13:40.494657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494504    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef4c9a2-e7df-46e3-8053-fde543232617-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ef4c9a2-e7df-46e3-8053-fde543232617" (UID: "9ef4c9a2-e7df-46e3-8053-fde543232617"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:13:40.494657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ef4c9a2-e7df-46e3-8053-fde543232617" (UID: "9ef4c9a2-e7df-46e3-8053-fde543232617"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.494880 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494662    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ef4c9a2-e7df-46e3-8053-fde543232617-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.494880 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494680    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.494880 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494683    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ef4c9a2-e7df-46e3-8053-fde543232617" (UID: "9ef4c9a2-e7df-46e3-8053-fde543232617"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.494880 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.494716    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-home" (OuterVolumeSpecName: "home") pod "9ef4c9a2-e7df-46e3-8053-fde543232617" (UID: "9ef4c9a2-e7df-46e3-8053-fde543232617"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.496676 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.496652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-dshm" (OuterVolumeSpecName: "dshm") pod "9ef4c9a2-e7df-46e3-8053-fde543232617" (UID: "9ef4c9a2-e7df-46e3-8053-fde543232617"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.496676 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.496662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef4c9a2-e7df-46e3-8053-fde543232617-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ef4c9a2-e7df-46e3-8053-fde543232617" (UID: "9ef4c9a2-e7df-46e3-8053-fde543232617"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:13:40.496865 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.496844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef4c9a2-e7df-46e3-8053-fde543232617-kube-api-access-7dm7n" (OuterVolumeSpecName: "kube-api-access-7dm7n") pod "9ef4c9a2-e7df-46e3-8053-fde543232617" (UID: "9ef4c9a2-e7df-46e3-8053-fde543232617"). InnerVolumeSpecName "kube-api-access-7dm7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:13:40.515716 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.515682    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ef4c9a2-e7df-46e3-8053-fde543232617" (UID: "9ef4c9a2-e7df-46e3-8053-fde543232617"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.595163 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.595133    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.595163 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.595161    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.595323 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.595170    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dm7n\" (UniqueName: \"kubernetes.io/projected/9ef4c9a2-e7df-46e3-8053-fde543232617-kube-api-access-7dm7n\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.595323 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.595181    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef4c9a2-e7df-46e3-8053-fde543232617-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.595323 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.595191    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.595323 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.595201    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef4c9a2-e7df-46e3-8053-fde543232617-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.727563 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.727534    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:13:40.796929 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.796899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-kserve-provision-location\") pod \"f604a7c4-28e1-430c-adf1-cc2be30fa464\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") "
Sep 16 20:13:40.797121 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.796943    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f604a7c4-28e1-430c-adf1-cc2be30fa464-cabundle-cert\") pod \"f604a7c4-28e1-430c-adf1-cc2be30fa464\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") "
Sep 16 20:13:40.797121 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.796985    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-tmp-dir\") pod \"f604a7c4-28e1-430c-adf1-cc2be30fa464\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") "
Sep 16 20:13:40.797121 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.797009    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f604a7c4-28e1-430c-adf1-cc2be30fa464-tls-certs\") pod \"f604a7c4-28e1-430c-adf1-cc2be30fa464\" (UID: \"f604a7c4-28e1-430c-adf1-cc2be30fa464\") "
Sep 16 20:13:40.797417 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.797357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f604a7c4-28e1-430c-adf1-cc2be30fa464-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f604a7c4-28e1-430c-adf1-cc2be30fa464" (UID: "f604a7c4-28e1-430c-adf1-cc2be30fa464"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:13:40.797500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.797481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f604a7c4-28e1-430c-adf1-cc2be30fa464" (UID: "f604a7c4-28e1-430c-adf1-cc2be30fa464"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.799161 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.799141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f604a7c4-28e1-430c-adf1-cc2be30fa464-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f604a7c4-28e1-430c-adf1-cc2be30fa464" (UID: "f604a7c4-28e1-430c-adf1-cc2be30fa464"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:13:40.818688 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.818662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f604a7c4-28e1-430c-adf1-cc2be30fa464" (UID: "f604a7c4-28e1-430c-adf1-cc2be30fa464"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.897966 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.897888    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.897966 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.897916    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f604a7c4-28e1-430c-adf1-cc2be30fa464-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.897966 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.897927    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f604a7c4-28e1-430c-adf1-cc2be30fa464-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.897966 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:40.897936    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f604a7c4-28e1-430c-adf1-cc2be30fa464-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:13:41.236583 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.236549    2561 generic.go:356] "Generic (PLEG): container finished" podID="f604a7c4-28e1-430c-adf1-cc2be30fa464" containerID="fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2" exitCode=0
Sep 16 20:13:41.237042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.236611    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"
Sep 16 20:13:41.237042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.236640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6" event={"ID":"f604a7c4-28e1-430c-adf1-cc2be30fa464","Type":"ContainerDied","Data":"fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2"}
Sep 16 20:13:41.237042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.236719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6" event={"ID":"f604a7c4-28e1-430c-adf1-cc2be30fa464","Type":"ContainerDied","Data":"0805d2efdeda071945190c0d508e40b00dfad47573d89d2e69d31c84e3e7888b"}
Sep 16 20:13:41.237042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.236747    2561 scope.go:117] "RemoveContainer" containerID="fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2"
Sep 16 20:13:41.238657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.238631    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerID="acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1" exitCode=0
Sep 16 20:13:41.238765 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.238678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" event={"ID":"9ef4c9a2-e7df-46e3-8053-fde543232617","Type":"ContainerDied","Data":"acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1"}
Sep 16 20:13:41.238765 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.238706    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"
Sep 16 20:13:41.238765 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.238720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg" event={"ID":"9ef4c9a2-e7df-46e3-8053-fde543232617","Type":"ContainerDied","Data":"67f9a44213ffdf04ba9f784215bfebe1c7055d53b057158c313e9b2625c9f601"}
Sep 16 20:13:41.245944 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.245922    2561 scope.go:117] "RemoveContainer" containerID="8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87"
Sep 16 20:13:41.254075 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.254059    2561 scope.go:117] "RemoveContainer" containerID="fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2"
Sep 16 20:13:41.254331 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:13:41.254313    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2\": container with ID starting with fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2 not found: ID does not exist" containerID="fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2"
Sep 16 20:13:41.254401 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.254339    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2"} err="failed to get container status \"fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2\": rpc error: code = NotFound desc = could not find container \"fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2\": container with ID starting with fc677b07479a43111d9d6b3a0aa91a2d1a31cc158d527883666e109d10f322f2 not found: ID does not exist"
Sep 16 20:13:41.254401 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.254355    2561 scope.go:117] "RemoveContainer" containerID="8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87"
Sep 16 20:13:41.254651 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:13:41.254631    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87\": container with ID starting with 8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87 not found: ID does not exist" containerID="8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87"
Sep 16 20:13:41.254718 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.254659    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87"} err="failed to get container status \"8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87\": rpc error: code = NotFound desc = could not find container \"8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87\": container with ID starting with 8852be754577b1381dddd8538664cbb15076ced914523f57c2afb2ecac7fbd87 not found: ID does not exist"
Sep 16 20:13:41.254718 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.254681    2561 scope.go:117] "RemoveContainer" containerID="acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1"
Sep 16 20:13:41.262311 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.262292    2561 scope.go:117] "RemoveContainer" containerID="f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3"
Sep 16 20:13:41.262988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.262969    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"]
Sep 16 20:13:41.266852 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.266825    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79f78f567d-4d4h6"]
Sep 16 20:13:41.269930 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.269912    2561 scope.go:117] "RemoveContainer" containerID="acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1"
Sep 16 20:13:41.270158 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:13:41.270142    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1\": container with ID starting with acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1 not found: ID does not exist" containerID="acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1"
Sep 16 20:13:41.270214 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.270184    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1"} err="failed to get container status \"acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1\": rpc error: code = NotFound desc = could not find container \"acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1\": container with ID starting with acf4739b8eb58abd68ed99963dac149b9492a189aff99919f3547f3e6831c2d1 not found: ID does not exist"
Sep 16 20:13:41.270214 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.270200    2561 scope.go:117] "RemoveContainer" containerID="f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3"
Sep 16 20:13:41.270464 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:13:41.270445    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3\": container with ID starting with f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3 not found: ID does not exist" containerID="f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3"
Sep 16 20:13:41.270537 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.270469    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3"} err="failed to get container status \"f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3\": rpc error: code = NotFound desc = could not find container \"f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3\": container with ID starting with f32e52f78c51f8f857d8a3bc2372cbed1ff5749f0d4f0f9767022374e5167fd3 not found: ID does not exist"
Sep 16 20:13:41.278492 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.278471    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"]
Sep 16 20:13:41.282456 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.282437    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-6wtsg"]
Sep 16 20:13:41.682415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.682356    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef4c9a2-e7df-46e3-8053-fde543232617" path="/var/lib/kubelet/pods/9ef4c9a2-e7df-46e3-8053-fde543232617/volumes"
Sep 16 20:13:41.682823 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:41.682808    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f604a7c4-28e1-430c-adf1-cc2be30fa464" path="/var/lib/kubelet/pods/f604a7c4-28e1-430c-adf1-cc2be30fa464/volumes"
Sep 16 20:13:48.652406 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:13:48.652362    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"]
Sep 16 20:14:03.671100 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.671057    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n" podUID="d0677295-7e57-4812-91a6-962b5d4a7e9f" containerName="main" containerID="cri-o://532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719" gracePeriod=45
Sep 16 20:14:03.916299 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.916273    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:14:03.952091 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.951998    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27hzf\" (UniqueName: \"kubernetes.io/projected/d0677295-7e57-4812-91a6-962b5d4a7e9f-kube-api-access-27hzf\") pod \"d0677295-7e57-4812-91a6-962b5d4a7e9f\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") "
Sep 16 20:14:03.952091 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.952040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0677295-7e57-4812-91a6-962b5d4a7e9f-tls-certs\") pod \"d0677295-7e57-4812-91a6-962b5d4a7e9f\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") "
Sep 16 20:14:03.952091 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.952060    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-dshm\") pod \"d0677295-7e57-4812-91a6-962b5d4a7e9f\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") "
Sep 16 20:14:03.952404 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.952103    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-tmp-dir\") pod \"d0677295-7e57-4812-91a6-962b5d4a7e9f\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") "
Sep 16 20:14:03.952404 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.952120    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-home\") pod \"d0677295-7e57-4812-91a6-962b5d4a7e9f\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") "
Sep 16 20:14:03.952404 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.952141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-model-cache\") pod \"d0677295-7e57-4812-91a6-962b5d4a7e9f\" (UID: \"d0677295-7e57-4812-91a6-962b5d4a7e9f\") "
Sep 16 20:14:03.952581 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.952482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0677295-7e57-4812-91a6-962b5d4a7e9f" (UID: "d0677295-7e57-4812-91a6-962b5d4a7e9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:14:03.952581 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.952488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-home" (OuterVolumeSpecName: "home") pod "d0677295-7e57-4812-91a6-962b5d4a7e9f" (UID: "d0677295-7e57-4812-91a6-962b5d4a7e9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:14:03.952581 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.952512    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0677295-7e57-4812-91a6-962b5d4a7e9f" (UID: "d0677295-7e57-4812-91a6-962b5d4a7e9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:14:03.954463 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.954435    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0677295-7e57-4812-91a6-962b5d4a7e9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0677295-7e57-4812-91a6-962b5d4a7e9f" (UID: "d0677295-7e57-4812-91a6-962b5d4a7e9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:14:03.954681 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.954662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-dshm" (OuterVolumeSpecName: "dshm") pod "d0677295-7e57-4812-91a6-962b5d4a7e9f" (UID: "d0677295-7e57-4812-91a6-962b5d4a7e9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:14:03.954681 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:03.954665    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0677295-7e57-4812-91a6-962b5d4a7e9f-kube-api-access-27hzf" (OuterVolumeSpecName: "kube-api-access-27hzf") pod "d0677295-7e57-4812-91a6-962b5d4a7e9f" (UID: "d0677295-7e57-4812-91a6-962b5d4a7e9f"). InnerVolumeSpecName "kube-api-access-27hzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:14:04.053002 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.052959    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:14:04.053002 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.052993    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:14:04.053002 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.053005    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27hzf\" (UniqueName: \"kubernetes.io/projected/d0677295-7e57-4812-91a6-962b5d4a7e9f-kube-api-access-27hzf\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:14:04.053002 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.053014    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0677295-7e57-4812-91a6-962b5d4a7e9f-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:14:04.053271 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.053022    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:14:04.053271 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.053029    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0677295-7e57-4812-91a6-962b5d4a7e9f-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:14:04.339424 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.339358    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0677295-7e57-4812-91a6-962b5d4a7e9f" containerID="532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719" exitCode=0
Sep 16 20:14:04.339424 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.339423    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"
Sep 16 20:14:04.339655 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.339430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n" event={"ID":"d0677295-7e57-4812-91a6-962b5d4a7e9f","Type":"ContainerDied","Data":"532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719"}
Sep 16 20:14:04.339655 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.339471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n" event={"ID":"d0677295-7e57-4812-91a6-962b5d4a7e9f","Type":"ContainerDied","Data":"1fa53133597bcdc4d697d3ab7c7f2d2941902ea697daf75227eac3e612f79772"}
Sep 16 20:14:04.339655 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.339494    2561 scope.go:117] "RemoveContainer" containerID="532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719"
Sep 16 20:14:04.347893 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.347872    2561 scope.go:117] "RemoveContainer" containerID="532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719"
Sep 16 20:14:04.348195 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:14:04.348173    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719\": container with ID starting with 532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719 not found: ID does not exist" containerID="532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719"
Sep 16 20:14:04.348243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.348205    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719"} err="failed to get container status \"532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719\": rpc error: code = NotFound desc = could not find container \"532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719\": container with ID starting with 532c07729e5fbec40d24ee24e5bf2a8712ffd5454d7f14e5ace8f27c72514719 not found: ID does not exist"
Sep 16 20:14:04.361335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.361305    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"]
Sep 16 20:14:04.364961 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:04.364934    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ntm9n"]
Sep 16 20:14:05.682274 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:05.682241    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0677295-7e57-4812-91a6-962b5d4a7e9f" path="/var/lib/kubelet/pods/d0677295-7e57-4812-91a6-962b5d4a7e9f/volumes"
Sep 16 20:14:13.749682 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:13.749655    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:14:13.754103 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:13.754081    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:14:32.908857 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.908818    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"]
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909090    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f604a7c4-28e1-430c-adf1-cc2be30fa464" containerName="main"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909104    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f604a7c4-28e1-430c-adf1-cc2be30fa464" containerName="main"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909116    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f604a7c4-28e1-430c-adf1-cc2be30fa464" containerName="storage-initializer"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909122    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f604a7c4-28e1-430c-adf1-cc2be30fa464" containerName="storage-initializer"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909133    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0677295-7e57-4812-91a6-962b5d4a7e9f" containerName="main"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909138    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0677295-7e57-4812-91a6-962b5d4a7e9f" containerName="main"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909145    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerName="main"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909151    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerName="main"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909156    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerName="storage-initializer"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909165    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerName="storage-initializer"
Sep 16 20:14:32.909243 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909236    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f604a7c4-28e1-430c-adf1-cc2be30fa464" containerName="main"
Sep 16 20:14:32.909632 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909251    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0677295-7e57-4812-91a6-962b5d4a7e9f" containerName="main"
Sep 16 20:14:32.909632 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.909261    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef4c9a2-e7df-46e3-8053-fde543232617" containerName="main"
Sep 16 20:14:32.913534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.913514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:32.917393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.917351    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 20:14:32.917393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.917361    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 20:14:32.917583 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.917435    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 20:14:32.917583 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.917527    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-jd96b\""
Sep 16 20:14:32.927461 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.927439    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"]
Sep 16 20:14:32.962178 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.962145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:32.962319 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.962182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-model-cache\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:32.962319 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.962210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-dshm\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:32.962319 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.962231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-home\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:32.962319 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.962280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bzl\" (UniqueName: \"kubernetes.io/projected/d4eebfad-9076-428f-abe7-4f2e3a3baa51-kube-api-access-52bzl\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:32.962502 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:32.962320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.063624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.063596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-dshm\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.063850 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.063628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-home\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.063850 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.063650    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bzl\" (UniqueName: \"kubernetes.io/projected/d4eebfad-9076-428f-abe7-4f2e3a3baa51-kube-api-access-52bzl\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.063850 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.063673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.063850 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.063715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.063850 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.063733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-model-cache\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.064123 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.064064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-model-cache\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.064181 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.064138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-home\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.064181 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.064166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.065826 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.065808    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-dshm\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.066098 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.066080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.073115 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.073095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bzl\" (UniqueName: \"kubernetes.io/projected/d4eebfad-9076-428f-abe7-4f2e3a3baa51-kube-api-access-52bzl\") pod \"decom-v1-kserve-6cb797fb75-qm642\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.223514 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.223426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:33.501492 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:33.501462    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"]
Sep 16 20:14:33.504308 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:14:33.504269    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4eebfad_9076_428f_abe7_4f2e3a3baa51.slice/crio-3202de5019e9bbe97632508ae8d20e2673ad6716d074e9d0028c6c5a72f56474 WatchSource:0}: Error finding container 3202de5019e9bbe97632508ae8d20e2673ad6716d074e9d0028c6c5a72f56474: Status 404 returned error can't find the container with id 3202de5019e9bbe97632508ae8d20e2673ad6716d074e9d0028c6c5a72f56474
Sep 16 20:14:34.464415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:34.464359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642" event={"ID":"d4eebfad-9076-428f-abe7-4f2e3a3baa51","Type":"ContainerStarted","Data":"8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da"}
Sep 16 20:14:34.464415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:34.464426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642" event={"ID":"d4eebfad-9076-428f-abe7-4f2e3a3baa51","Type":"ContainerStarted","Data":"3202de5019e9bbe97632508ae8d20e2673ad6716d074e9d0028c6c5a72f56474"}
Sep 16 20:14:34.487595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:34.487546    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642" podStartSLOduration=2.487531786 podStartE2EDuration="2.487531786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:14:34.486453177 +0000 UTC m=+2121.258323587" watchObservedRunningTime="2026-09-16 20:14:34.487531786 +0000 UTC m=+2121.259402196"
Sep 16 20:14:35.223809 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:35.223757    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:43.224571 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:43.224529    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:43.237095 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:43.237070    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:14:43.512367 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:14:43.512339    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:15:01.893433 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.893391    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"]
Sep 16 20:15:01.896509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.896483    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:01.900115 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.900086    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 20:15:01.900261 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.900151    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 20:15:01.900261 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.900188    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:15:01.901262 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.901241    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 20:15:01.901412 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.901290    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 20:15:01.901412 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.901359    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-bn8cc\""
Sep 16 20:15:01.912877 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.912846    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"]
Sep 16 20:15:01.981279 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.981245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-model-cache\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:01.981488 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.981288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxjb\" (UniqueName: \"kubernetes.io/projected/ea2b0866-24e7-4e10-b060-1074f057eb4b-kube-api-access-cpxjb\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:01.981488 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.981312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2b0866-24e7-4e10-b060-1074f057eb4b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:01.981488 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.981426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2b0866-24e7-4e10-b060-1074f057eb4b-tls-certs\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:01.981488 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.981458    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-home\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:01.981642 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.981511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-dshm\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:01.981642 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.981533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:01.981642 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:01.981554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.081958 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.081916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-model-cache\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082159 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.081967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxjb\" (UniqueName: \"kubernetes.io/projected/ea2b0866-24e7-4e10-b060-1074f057eb4b-kube-api-access-cpxjb\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082159 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.081989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2b0866-24e7-4e10-b060-1074f057eb4b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082159 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2b0866-24e7-4e10-b060-1074f057eb4b-tls-certs\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082159 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-home\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082159 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-dshm\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082159 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082159 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082574 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082660 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082574    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-model-cache\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082660 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-home\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.082755 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.082665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.083059 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.083036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2b0866-24e7-4e10-b060-1074f057eb4b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.084304 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.084281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-dshm\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.084872 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.084849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2b0866-24e7-4e10-b060-1074f057eb4b-tls-certs\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.093844 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.093811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxjb\" (UniqueName: \"kubernetes.io/projected/ea2b0866-24e7-4e10-b060-1074f057eb4b-kube-api-access-cpxjb\") pod \"tokenizer-template-args-test-kserve-54d4855974-629r6\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.144233 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.144145    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"]
Sep 16 20:15:02.148888 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.148860    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.158181 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.158154    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"]
Sep 16 20:15:02.207540 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.207500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:02.282969 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.282938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.282969 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.282976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.283166 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.283023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.283166 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.283101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.384286 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.384246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.384496 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.384299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.384496 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.384334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.384496 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.384409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.384820 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.384799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.384863 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.384830    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.385059 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.385039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.386925 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.386908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.460045 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.459959    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:02.481420 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.481393    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"]
Sep 16 20:15:02.483919 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:15:02.483886    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2b0866_24e7_4e10_b060_1074f057eb4b.slice/crio-ef1b901404ba5e3c12cb53da5d911a27e5d1ef2da6a68b2e2f679d99c3a5c495 WatchSource:0}: Error finding container ef1b901404ba5e3c12cb53da5d911a27e5d1ef2da6a68b2e2f679d99c3a5c495: Status 404 returned error can't find the container with id ef1b901404ba5e3c12cb53da5d911a27e5d1ef2da6a68b2e2f679d99c3a5c495
Sep 16 20:15:02.579455 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.579422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" event={"ID":"ea2b0866-24e7-4e10-b060-1074f057eb4b","Type":"ContainerStarted","Data":"ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9"}
Sep 16 20:15:02.579650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.579474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" event={"ID":"ea2b0866-24e7-4e10-b060-1074f057eb4b","Type":"ContainerStarted","Data":"ef1b901404ba5e3c12cb53da5d911a27e5d1ef2da6a68b2e2f679d99c3a5c495"}
Sep 16 20:15:02.741430 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:02.741401    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"]
Sep 16 20:15:02.743705 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:15:02.743680    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5bcf0d_9631_45d1_bf75_7955eb0f3183.slice/crio-9cda9317a68e0ec4a2ca9e41c12d964b523c3d7cba7e20452fda0cc785d7f03a WatchSource:0}: Error finding container 9cda9317a68e0ec4a2ca9e41c12d964b523c3d7cba7e20452fda0cc785d7f03a: Status 404 returned error can't find the container with id 9cda9317a68e0ec4a2ca9e41c12d964b523c3d7cba7e20452fda0cc785d7f03a
Sep 16 20:15:03.590646 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:03.590607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h" event={"ID":"cf5bcf0d-9631-45d1-bf75-7955eb0f3183","Type":"ContainerStarted","Data":"91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0"}
Sep 16 20:15:03.591054 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:03.590693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h" event={"ID":"cf5bcf0d-9631-45d1-bf75-7955eb0f3183","Type":"ContainerStarted","Data":"9cda9317a68e0ec4a2ca9e41c12d964b523c3d7cba7e20452fda0cc785d7f03a"}
Sep 16 20:15:05.602090 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:05.602057    2561 generic.go:356] "Generic (PLEG): container finished" podID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerID="ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9" exitCode=0
Sep 16 20:15:05.602589 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:05.602133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" event={"ID":"ea2b0866-24e7-4e10-b060-1074f057eb4b","Type":"ContainerDied","Data":"ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9"}
Sep 16 20:15:05.603767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:05.603744    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" containerID="91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0" exitCode=0
Sep 16 20:15:05.603904 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:05.603797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h" event={"ID":"cf5bcf0d-9631-45d1-bf75-7955eb0f3183","Type":"ContainerDied","Data":"91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0"}
Sep 16 20:15:06.610614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:06.610523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" event={"ID":"ea2b0866-24e7-4e10-b060-1074f057eb4b","Type":"ContainerStarted","Data":"a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59"}
Sep 16 20:15:06.612512 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:06.612486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h" event={"ID":"cf5bcf0d-9631-45d1-bf75-7955eb0f3183","Type":"ContainerStarted","Data":"63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467"}
Sep 16 20:15:06.612729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:06.612712    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:06.631501 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:06.631447    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" podStartSLOduration=3.033032615 podStartE2EDuration="5.631430375s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.59839776s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:15:06.629980083 +0000 UTC m=+2153.401850495" watchObservedRunningTime="2026-09-16 20:15:06.631430375 +0000 UTC m=+2153.403300786"
Sep 16 20:15:06.649343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:06.649287    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h" podStartSLOduration=2.077849846 podStartE2EDuration="4.649271031s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.571421185s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:15:06.648398376 +0000 UTC m=+2153.420268788" watchObservedRunningTime="2026-09-16 20:15:06.649271031 +0000 UTC m=+2153.421141443"
Sep 16 20:15:07.208521 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:07.208484    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:12.207920 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:12.207874    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:12.220676 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:12.220646    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:12.651963 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:12.651931    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:37.631016 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:37.630978    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:40.526142 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:40.526111    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h_cf5bcf0d-9631-45d1-bf75-7955eb0f3183/main/0.log"
Sep 16 20:15:40.917230 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:40.917149    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"]
Sep 16 20:15:40.917411 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:40.917343    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h" podUID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" containerName="main" containerID="cri-o://63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467" gracePeriod=30
Sep 16 20:15:40.944475 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:40.944434    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"]
Sep 16 20:15:40.953443 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:15:40.953410    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:15:40.953443 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:15:40.953443 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:15:40.953443 ip-10-0-136-120 kubenswrapper[2561]:  > containerID="a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59" cmd=["/bin/sleep","15"]
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:40.953459    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" message="" err=<
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]:  >
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:15:40.953471    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" podUID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerName="main" containerID="cri-o://a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59"
Sep 16 20:15:40.953678 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:40.953508    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" podUID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerName="main" containerID="cri-o://a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59" gracePeriod=60
Sep 16 20:15:41.306988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.306958    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:41.415662 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-model-cache\") pod \"ea2b0866-24e7-4e10-b060-1074f057eb4b\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") "
Sep 16 20:15:41.415662 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415665    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-kserve-provision-location\") pod \"ea2b0866-24e7-4e10-b060-1074f057eb4b\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") "
Sep 16 20:15:41.415892 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415687    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpxjb\" (UniqueName: \"kubernetes.io/projected/ea2b0866-24e7-4e10-b060-1074f057eb4b-kube-api-access-cpxjb\") pod \"ea2b0866-24e7-4e10-b060-1074f057eb4b\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") "
Sep 16 20:15:41.415892 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415706    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2b0866-24e7-4e10-b060-1074f057eb4b-tls-certs\") pod \"ea2b0866-24e7-4e10-b060-1074f057eb4b\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") "
Sep 16 20:15:41.415892 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415741    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-home\") pod \"ea2b0866-24e7-4e10-b060-1074f057eb4b\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") "
Sep 16 20:15:41.415892 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-dshm\") pod \"ea2b0866-24e7-4e10-b060-1074f057eb4b\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") "
Sep 16 20:15:41.415892 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-tmp-dir\") pod \"ea2b0866-24e7-4e10-b060-1074f057eb4b\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") "
Sep 16 20:15:41.415892 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415827    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2b0866-24e7-4e10-b060-1074f057eb4b-cabundle-cert\") pod \"ea2b0866-24e7-4e10-b060-1074f057eb4b\" (UID: \"ea2b0866-24e7-4e10-b060-1074f057eb4b\") "
Sep 16 20:15:41.416214 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.415917    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea2b0866-24e7-4e10-b060-1074f057eb4b" (UID: "ea2b0866-24e7-4e10-b060-1074f057eb4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.416214 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.416045    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-home" (OuterVolumeSpecName: "home") pod "ea2b0866-24e7-4e10-b060-1074f057eb4b" (UID: "ea2b0866-24e7-4e10-b060-1074f057eb4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.416214 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.416060    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.416357 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.416218    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea2b0866-24e7-4e10-b060-1074f057eb4b" (UID: "ea2b0866-24e7-4e10-b060-1074f057eb4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.416357 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.416291    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2b0866-24e7-4e10-b060-1074f057eb4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea2b0866-24e7-4e10-b060-1074f057eb4b" (UID: "ea2b0866-24e7-4e10-b060-1074f057eb4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:15:41.418049 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.417936    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2b0866-24e7-4e10-b060-1074f057eb4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea2b0866-24e7-4e10-b060-1074f057eb4b" (UID: "ea2b0866-24e7-4e10-b060-1074f057eb4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:15:41.418049 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.418017    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2b0866-24e7-4e10-b060-1074f057eb4b-kube-api-access-cpxjb" (OuterVolumeSpecName: "kube-api-access-cpxjb") pod "ea2b0866-24e7-4e10-b060-1074f057eb4b" (UID: "ea2b0866-24e7-4e10-b060-1074f057eb4b"). InnerVolumeSpecName "kube-api-access-cpxjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:15:41.418049 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.418025    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-dshm" (OuterVolumeSpecName: "dshm") pod "ea2b0866-24e7-4e10-b060-1074f057eb4b" (UID: "ea2b0866-24e7-4e10-b060-1074f057eb4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.445282 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.445239    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea2b0866-24e7-4e10-b060-1074f057eb4b" (UID: "ea2b0866-24e7-4e10-b060-1074f057eb4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.516976 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.516943    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.516976 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.516971    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxjb\" (UniqueName: \"kubernetes.io/projected/ea2b0866-24e7-4e10-b060-1074f057eb4b-kube-api-access-cpxjb\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.516976 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.516982    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2b0866-24e7-4e10-b060-1074f057eb4b-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.517193 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.516992    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.517193 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.517000    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.517193 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.517008    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2b0866-24e7-4e10-b060-1074f057eb4b-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.517193 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.517015    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2b0866-24e7-4e10-b060-1074f057eb4b-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.557619 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.557593    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:41.617638 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.617600    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tmp-dir\") pod \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") "
Sep 16 20:15:41.617812 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.617654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-kserve-provision-location\") pod \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") "
Sep 16 20:15:41.617812 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.617682    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-cabundle-cert\") pod \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") "
Sep 16 20:15:41.617812 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.617723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tls-certs\") pod \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\" (UID: \"cf5bcf0d-9631-45d1-bf75-7955eb0f3183\") "
Sep 16 20:15:41.618095 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.618069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf5bcf0d-9631-45d1-bf75-7955eb0f3183" (UID: "cf5bcf0d-9631-45d1-bf75-7955eb0f3183"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.618134 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.618067    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf5bcf0d-9631-45d1-bf75-7955eb0f3183" (UID: "cf5bcf0d-9631-45d1-bf75-7955eb0f3183"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:15:41.619837 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.619818    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf5bcf0d-9631-45d1-bf75-7955eb0f3183" (UID: "cf5bcf0d-9631-45d1-bf75-7955eb0f3183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:15:41.639423 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.639342    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf5bcf0d-9631-45d1-bf75-7955eb0f3183" (UID: "cf5bcf0d-9631-45d1-bf75-7955eb0f3183"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.718814 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.718785    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.718814 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.718812    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.718814 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.718823    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.719026 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.718831    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5bcf0d-9631-45d1-bf75-7955eb0f3183-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.772002 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.771958    2561 generic.go:356] "Generic (PLEG): container finished" podID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerID="a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59" exitCode=0
Sep 16 20:15:41.772174 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.772012    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"
Sep 16 20:15:41.772174 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.772050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" event={"ID":"ea2b0866-24e7-4e10-b060-1074f057eb4b","Type":"ContainerDied","Data":"a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59"}
Sep 16 20:15:41.772174 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.772122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6" event={"ID":"ea2b0866-24e7-4e10-b060-1074f057eb4b","Type":"ContainerDied","Data":"ef1b901404ba5e3c12cb53da5d911a27e5d1ef2da6a68b2e2f679d99c3a5c495"}
Sep 16 20:15:41.772290 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.772157    2561 scope.go:117] "RemoveContainer" containerID="a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59"
Sep 16 20:15:41.773846 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.773811    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" containerID="63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467" exitCode=0
Sep 16 20:15:41.773990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.773862    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"
Sep 16 20:15:41.773990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.773890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h" event={"ID":"cf5bcf0d-9631-45d1-bf75-7955eb0f3183","Type":"ContainerDied","Data":"63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467"}
Sep 16 20:15:41.773990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.773935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h" event={"ID":"cf5bcf0d-9631-45d1-bf75-7955eb0f3183","Type":"ContainerDied","Data":"9cda9317a68e0ec4a2ca9e41c12d964b523c3d7cba7e20452fda0cc785d7f03a"}
Sep 16 20:15:41.785270 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.785246    2561 scope.go:117] "RemoveContainer" containerID="ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9"
Sep 16 20:15:41.792671 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.792652    2561 scope.go:117] "RemoveContainer" containerID="a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59"
Sep 16 20:15:41.792941 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:15:41.792914    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59\": container with ID starting with a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59 not found: ID does not exist" containerID="a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59"
Sep 16 20:15:41.793000 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.792941    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59"} err="failed to get container status \"a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59\": rpc error: code = NotFound desc = could not find container \"a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59\": container with ID starting with a33e51e287f3406f153b9e5b193dfefb54c99c4bf8a5b7bb6bf1b351db6b2c59 not found: ID does not exist"
Sep 16 20:15:41.793000 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.792957    2561 scope.go:117] "RemoveContainer" containerID="ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9"
Sep 16 20:15:41.793253 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:15:41.793232    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9\": container with ID starting with ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9 not found: ID does not exist" containerID="ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9"
Sep 16 20:15:41.793293 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.793259    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9"} err="failed to get container status \"ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9\": rpc error: code = NotFound desc = could not find container \"ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9\": container with ID starting with ab40efd642d0d979f3bb21bbd3e30a8dcb0377d5867dcc8544691f41857e45c9 not found: ID does not exist"
Sep 16 20:15:41.793293 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.793275    2561 scope.go:117] "RemoveContainer" containerID="63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467"
Sep 16 20:15:41.799326 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.799306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"]
Sep 16 20:15:41.801418 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.801354    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-629r6"]
Sep 16 20:15:41.802078 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.802063    2561 scope.go:117] "RemoveContainer" containerID="91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0"
Sep 16 20:15:41.809296 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.809279    2561 scope.go:117] "RemoveContainer" containerID="63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467"
Sep 16 20:15:41.809620 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:15:41.809593    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467\": container with ID starting with 63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467 not found: ID does not exist" containerID="63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467"
Sep 16 20:15:41.809688 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.809621    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467"} err="failed to get container status \"63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467\": rpc error: code = NotFound desc = could not find container \"63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467\": container with ID starting with 63940969d796250233709fef733e06efdb8a80b11e222cd0ce66063f1a2c6467 not found: ID does not exist"
Sep 16 20:15:41.809688 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.809640    2561 scope.go:117] "RemoveContainer" containerID="91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0"
Sep 16 20:15:41.809887 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:15:41.809869    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0\": container with ID starting with 91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0 not found: ID does not exist" containerID="91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0"
Sep 16 20:15:41.809932 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.809894    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0"} err="failed to get container status \"91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0\": rpc error: code = NotFound desc = could not find container \"91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0\": container with ID starting with 91dab6b88e307ea8a18fe3f6965350a73e0e8b3fa9d8f08d1b2f6ea560b5d0f0 not found: ID does not exist"
Sep 16 20:15:41.812927 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.812905    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"]
Sep 16 20:15:41.816629 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:41.816607    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85d5559b66-pbv2h"]
Sep 16 20:15:43.682605 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:43.682572    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" path="/var/lib/kubelet/pods/cf5bcf0d-9631-45d1-bf75-7955eb0f3183/volumes"
Sep 16 20:15:43.683029 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:43.683016    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2b0866-24e7-4e10-b060-1074f057eb4b" path="/var/lib/kubelet/pods/ea2b0866-24e7-4e10-b060-1074f057eb4b/volumes"
Sep 16 20:15:53.632531 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:15:53.632490    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"]
Sep 16 20:16:08.651119 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:08.651057    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642" podUID="d4eebfad-9076-428f-abe7-4f2e3a3baa51" containerName="main" containerID="cri-o://8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da" gracePeriod=45
Sep 16 20:16:08.890764 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:08.890736    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:16:08.891760 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:08.891734    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4eebfad-9076-428f-abe7-4f2e3a3baa51" containerID="8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da" exitCode=0
Sep 16 20:16:08.891883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:08.891780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642" event={"ID":"d4eebfad-9076-428f-abe7-4f2e3a3baa51","Type":"ContainerDied","Data":"8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da"}
Sep 16 20:16:08.891883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:08.891821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642" event={"ID":"d4eebfad-9076-428f-abe7-4f2e3a3baa51","Type":"ContainerDied","Data":"3202de5019e9bbe97632508ae8d20e2673ad6716d074e9d0028c6c5a72f56474"}
Sep 16 20:16:08.891883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:08.891849    2561 scope.go:117] "RemoveContainer" containerID="8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da"
Sep 16 20:16:08.899381 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:08.899350    2561 scope.go:117] "RemoveContainer" containerID="8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da"
Sep 16 20:16:08.899654 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:16:08.899634    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da\": container with ID starting with 8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da not found: ID does not exist" containerID="8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da"
Sep 16 20:16:08.899695 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:08.899662    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da"} err="failed to get container status \"8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da\": rpc error: code = NotFound desc = could not find container \"8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da\": container with ID starting with 8314823903f69c734b3b5dd9be538058faa7874fbd42d93b960ec3777df380da not found: ID does not exist"
Sep 16 20:16:09.034181 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-home\") pod \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") "
Sep 16 20:16:09.034181 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034184    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tls-certs\") pod \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") "
Sep 16 20:16:09.034440 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52bzl\" (UniqueName: \"kubernetes.io/projected/d4eebfad-9076-428f-abe7-4f2e3a3baa51-kube-api-access-52bzl\") pod \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") "
Sep 16 20:16:09.034440 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034247    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-dshm\") pod \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") "
Sep 16 20:16:09.034440 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034300    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-model-cache\") pod \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") "
Sep 16 20:16:09.034440 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034348    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tmp-dir\") pod \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\" (UID: \"d4eebfad-9076-428f-abe7-4f2e3a3baa51\") "
Sep 16 20:16:09.034644 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034490    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-home" (OuterVolumeSpecName: "home") pod "d4eebfad-9076-428f-abe7-4f2e3a3baa51" (UID: "d4eebfad-9076-428f-abe7-4f2e3a3baa51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:09.034644 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034551    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:16:09.034743 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034622    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4eebfad-9076-428f-abe7-4f2e3a3baa51" (UID: "d4eebfad-9076-428f-abe7-4f2e3a3baa51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:09.034743 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.034674    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4eebfad-9076-428f-abe7-4f2e3a3baa51" (UID: "d4eebfad-9076-428f-abe7-4f2e3a3baa51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:09.036645 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.036620    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4eebfad-9076-428f-abe7-4f2e3a3baa51" (UID: "d4eebfad-9076-428f-abe7-4f2e3a3baa51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:16:09.036752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.036670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-dshm" (OuterVolumeSpecName: "dshm") pod "d4eebfad-9076-428f-abe7-4f2e3a3baa51" (UID: "d4eebfad-9076-428f-abe7-4f2e3a3baa51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:09.036752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.036678    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4eebfad-9076-428f-abe7-4f2e3a3baa51-kube-api-access-52bzl" (OuterVolumeSpecName: "kube-api-access-52bzl") pod "d4eebfad-9076-428f-abe7-4f2e3a3baa51" (UID: "d4eebfad-9076-428f-abe7-4f2e3a3baa51"). InnerVolumeSpecName "kube-api-access-52bzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:16:09.135335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.135306    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:16:09.135335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.135331    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:16:09.135335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.135339    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eebfad-9076-428f-abe7-4f2e3a3baa51-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:16:09.135581 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.135347    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52bzl\" (UniqueName: \"kubernetes.io/projected/d4eebfad-9076-428f-abe7-4f2e3a3baa51-kube-api-access-52bzl\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:16:09.135581 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.135355    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eebfad-9076-428f-abe7-4f2e3a3baa51-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:16:09.896831 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.896797    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"
Sep 16 20:16:09.915387 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.915346    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"]
Sep 16 20:16:09.919399 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:09.919359    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qm642"]
Sep 16 20:16:11.682839 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:11.682805    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4eebfad-9076-428f-abe7-4f2e3a3baa51" path="/var/lib/kubelet/pods/d4eebfad-9076-428f-abe7-4f2e3a3baa51/volumes"
Sep 16 20:16:13.086491 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086451    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"]
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086706    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerName="storage-initializer"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086717    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerName="storage-initializer"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086727    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" containerName="storage-initializer"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086733    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" containerName="storage-initializer"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086742    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" containerName="main"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086747    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" containerName="main"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086761    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerName="main"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086766    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerName="main"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086773    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4eebfad-9076-428f-abe7-4f2e3a3baa51" containerName="main"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086778    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4eebfad-9076-428f-abe7-4f2e3a3baa51" containerName="main"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086836    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5bcf0d-9631-45d1-bf75-7955eb0f3183" containerName="main"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086843    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2b0866-24e7-4e10-b060-1074f057eb4b" containerName="main"
Sep 16 20:16:13.086954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.086851    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4eebfad-9076-428f-abe7-4f2e3a3baa51" containerName="main"
Sep 16 20:16:13.090990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.090967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.095027 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.095002    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 20:16:13.095027 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.095014    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-djmhg\""
Sep 16 20:16:13.095214 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.095122    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 20:16:13.095318 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.095300    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 20:16:13.100629 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.100601    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"]
Sep 16 20:16:13.166257 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.166217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-model-cache\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.166466 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.166265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-home\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.166466 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.166294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.166466 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.166352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ltp\" (UniqueName: \"kubernetes.io/projected/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-kube-api-access-w7ltp\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.166466 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.166411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tls-certs\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.166466 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.166466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-dshm\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.267558 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.267521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tls-certs\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.267693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.267574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-dshm\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.267693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.267608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-model-cache\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.267693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.267629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-home\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.267693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.267646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.267693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.267664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ltp\" (UniqueName: \"kubernetes.io/projected/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-kube-api-access-w7ltp\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.268024 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.268001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-model-cache\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.268095 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.268066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.268145 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.268128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-home\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.269917 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.269895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-dshm\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.270274 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.270254    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tls-certs\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.275837 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.275812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ltp\" (UniqueName: \"kubernetes.io/projected/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-kube-api-access-w7ltp\") pod \"leave-v1-kserve-65684d4f5-jcqvd\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.402004 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.401923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:13.684184 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.684152    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"]
Sep 16 20:16:13.684713 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:16:13.684686    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e8da7a_5279_4bf4_8e27_a2a5f96c053a.slice/crio-11d2a7c39e910763fa86523aec619018c11616701ccbce3e743109b93556cb0b WatchSource:0}: Error finding container 11d2a7c39e910763fa86523aec619018c11616701ccbce3e743109b93556cb0b: Status 404 returned error can't find the container with id 11d2a7c39e910763fa86523aec619018c11616701ccbce3e743109b93556cb0b
Sep 16 20:16:13.915628 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.915589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd" event={"ID":"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a","Type":"ContainerStarted","Data":"32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab"}
Sep 16 20:16:13.915823 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.915657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd" event={"ID":"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a","Type":"ContainerStarted","Data":"11d2a7c39e910763fa86523aec619018c11616701ccbce3e743109b93556cb0b"}
Sep 16 20:16:13.937152 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:13.937054    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd" podStartSLOduration=0.937037807 podStartE2EDuration="937.037807ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:16:13.935726368 +0000 UTC m=+2220.707596779" watchObservedRunningTime="2026-09-16 20:16:13.937037807 +0000 UTC m=+2220.708908217"
Sep 16 20:16:14.402477 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:14.402442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:23.403136 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:23.403091    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:23.415706 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:23.415682    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:16:23.970453 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:16:23.970423    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:17:00.790931 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.790888    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"]
Sep 16 20:17:00.794076 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.794049    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.798228 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.798201    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 20:17:00.798412 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.798307    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-5p92g\""
Sep 16 20:17:00.798412 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.798338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 20:17:00.798588 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.798570    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 20:17:00.803259 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.803215    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"]
Sep 16 20:17:00.824943 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.824913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a203d7c-e245-4812-b7e8-3517c39f0954-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.825091 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.824959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.825091 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.824979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.825091 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.824993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.825091 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.825010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.825091 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.825064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m62vv\" (UniqueName: \"kubernetes.io/projected/4a203d7c-e245-4812-b7e8-3517c39f0954-kube-api-access-m62vv\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.925738 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.925699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a203d7c-e245-4812-b7e8-3517c39f0954-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.925939 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.925766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.925939 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.925800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.925939 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.925827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.926115 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.925964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.926115 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.926009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62vv\" (UniqueName: \"kubernetes.io/projected/4a203d7c-e245-4812-b7e8-3517c39f0954-kube-api-access-m62vv\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.926311 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.926284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.926311 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.926303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.926476 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.926354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.928079 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.928055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.928280 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.928262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a203d7c-e245-4812-b7e8-3517c39f0954-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:00.934215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:00.934192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62vv\" (UniqueName: \"kubernetes.io/projected/4a203d7c-e245-4812-b7e8-3517c39f0954-kube-api-access-m62vv\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:01.105621 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:01.105528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:01.379972 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:01.379946    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"]
Sep 16 20:17:01.382519 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:17:01.382489    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a203d7c_e245_4812_b7e8_3517c39f0954.slice/crio-115d40848465687fc02c172ba21d395de2a4ff85e982cf7bd5edd0237500effa WatchSource:0}: Error finding container 115d40848465687fc02c172ba21d395de2a4ff85e982cf7bd5edd0237500effa: Status 404 returned error can't find the container with id 115d40848465687fc02c172ba21d395de2a4ff85e982cf7bd5edd0237500effa
Sep 16 20:17:02.120052 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:02.120016    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" event={"ID":"4a203d7c-e245-4812-b7e8-3517c39f0954","Type":"ContainerStarted","Data":"7326a68f849bd4d2d056ec9903c0a1416423c2477faedc68329b515928b80f01"}
Sep 16 20:17:02.120602 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:02.120078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" event={"ID":"4a203d7c-e245-4812-b7e8-3517c39f0954","Type":"ContainerStarted","Data":"115d40848465687fc02c172ba21d395de2a4ff85e982cf7bd5edd0237500effa"}
Sep 16 20:17:02.142675 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:02.142618    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" podStartSLOduration=2.142600923 podStartE2EDuration="2.142600923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:17:02.140498639 +0000 UTC m=+2268.912369049" watchObservedRunningTime="2026-09-16 20:17:02.142600923 +0000 UTC m=+2268.914471333"
Sep 16 20:17:03.106354 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:03.106308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:11.106356 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:11.106320    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:11.118763 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:11.118730    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:11.167413 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:11.167363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:17:35.018850 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:35.018806    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"]
Sep 16 20:17:50.038112 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.038049    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd" podUID="c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" containerName="main" containerID="cri-o://32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab" gracePeriod=45
Sep 16 20:17:50.279449 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.279424    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:17:50.317568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.317489    2561 generic.go:356] "Generic (PLEG): container finished" podID="c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" containerID="32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab" exitCode=0
Sep 16 20:17:50.317568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.317532    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"
Sep 16 20:17:50.317727 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.317579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd" event={"ID":"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a","Type":"ContainerDied","Data":"32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab"}
Sep 16 20:17:50.317727 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.317628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd" event={"ID":"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a","Type":"ContainerDied","Data":"11d2a7c39e910763fa86523aec619018c11616701ccbce3e743109b93556cb0b"}
Sep 16 20:17:50.317727 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.317652    2561 scope.go:117] "RemoveContainer" containerID="32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab"
Sep 16 20:17:50.326078 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.326053    2561 scope.go:117] "RemoveContainer" containerID="32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab"
Sep 16 20:17:50.326410 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:17:50.326387    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab\": container with ID starting with 32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab not found: ID does not exist" containerID="32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab"
Sep 16 20:17:50.326479 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.326423    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab"} err="failed to get container status \"32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab\": rpc error: code = NotFound desc = could not find container \"32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab\": container with ID starting with 32191fb507768c55e64aa279619254efd9802d6254a6b833519741a1ebfaedab not found: ID does not exist"
Sep 16 20:17:50.424956 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.424925    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7ltp\" (UniqueName: \"kubernetes.io/projected/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-kube-api-access-w7ltp\") pod \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") "
Sep 16 20:17:50.425120 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.424964    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tls-certs\") pod \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") "
Sep 16 20:17:50.425120 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.424987    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-model-cache\") pod \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") "
Sep 16 20:17:50.425120 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.425018    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tmp-dir\") pod \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") "
Sep 16 20:17:50.425120 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.425070    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-home\") pod \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") "
Sep 16 20:17:50.425120 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.425104    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-dshm\") pod \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\" (UID: \"c2e8da7a-5279-4bf4-8e27-a2a5f96c053a\") "
Sep 16 20:17:50.425415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.425271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" (UID: "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:17:50.425415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.425311    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" (UID: "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:17:50.425415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.425397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-home" (OuterVolumeSpecName: "home") pod "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" (UID: "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:17:50.427289 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.427265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-dshm" (OuterVolumeSpecName: "dshm") pod "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" (UID: "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:17:50.427618 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.427598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" (UID: "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:17:50.427720 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.427617    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-kube-api-access-w7ltp" (OuterVolumeSpecName: "kube-api-access-w7ltp") pod "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" (UID: "c2e8da7a-5279-4bf4-8e27-a2a5f96c053a"). InnerVolumeSpecName "kube-api-access-w7ltp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:17:50.526196 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.526167    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:17:50.526196 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.526195    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ltp\" (UniqueName: \"kubernetes.io/projected/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-kube-api-access-w7ltp\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:17:50.526414 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.526211    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:17:50.526414 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.526223    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:17:50.526414 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.526233    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:17:50.526414 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.526243    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:17:50.640131 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.640099    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"]
Sep 16 20:17:50.644122 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:50.644091    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-jcqvd"]
Sep 16 20:17:51.682793 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:17:51.682705    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" path="/var/lib/kubelet/pods/c2e8da7a-5279-4bf4-8e27-a2a5f96c053a/volumes"
Sep 16 20:18:06.054737 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.054703    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"]
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:18:06.063477    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:  > containerID="7326a68f849bd4d2d056ec9903c0a1416423c2477faedc68329b515928b80f01" cmd=["/bin/sleep","15"]
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.063527    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" message="" err=<
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:  >
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:18:06.063539    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:18:06.063543 ip-10-0-136-120 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" podUID="4a203d7c-e245-4812-b7e8-3517c39f0954" containerName="main" containerID="cri-o://7326a68f849bd4d2d056ec9903c0a1416423c2477faedc68329b515928b80f01"
Sep 16 20:18:06.064072 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.063587    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" podUID="4a203d7c-e245-4812-b7e8-3517c39f0954" containerName="main" containerID="cri-o://7326a68f849bd4d2d056ec9903c0a1416423c2477faedc68329b515928b80f01" gracePeriod=60
Sep 16 20:18:06.156580 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.156535    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" podUID="4a203d7c-e245-4812-b7e8-3517c39f0954" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 16 20:18:06.384428 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.384393    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a203d7c-e245-4812-b7e8-3517c39f0954" containerID="7326a68f849bd4d2d056ec9903c0a1416423c2477faedc68329b515928b80f01" exitCode=0
Sep 16 20:18:06.384607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.384403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" event={"ID":"4a203d7c-e245-4812-b7e8-3517c39f0954","Type":"ContainerDied","Data":"7326a68f849bd4d2d056ec9903c0a1416423c2477faedc68329b515928b80f01"}
Sep 16 20:18:06.410104 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.410079    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:18:06.456640 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456605    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-model-cache\") pod \"4a203d7c-e245-4812-b7e8-3517c39f0954\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") "
Sep 16 20:18:06.456802 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-home\") pod \"4a203d7c-e245-4812-b7e8-3517c39f0954\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") "
Sep 16 20:18:06.456802 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456693    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-tmp-dir\") pod \"4a203d7c-e245-4812-b7e8-3517c39f0954\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") "
Sep 16 20:18:06.456802 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a203d7c-e245-4812-b7e8-3517c39f0954-tls-certs\") pod \"4a203d7c-e245-4812-b7e8-3517c39f0954\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") "
Sep 16 20:18:06.456802 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456754    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-dshm\") pod \"4a203d7c-e245-4812-b7e8-3517c39f0954\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") "
Sep 16 20:18:06.456802 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456783    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m62vv\" (UniqueName: \"kubernetes.io/projected/4a203d7c-e245-4812-b7e8-3517c39f0954-kube-api-access-m62vv\") pod \"4a203d7c-e245-4812-b7e8-3517c39f0954\" (UID: \"4a203d7c-e245-4812-b7e8-3517c39f0954\") "
Sep 16 20:18:06.457036 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456874    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a203d7c-e245-4812-b7e8-3517c39f0954" (UID: "4a203d7c-e245-4812-b7e8-3517c39f0954"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:18:06.457036 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456986    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-home" (OuterVolumeSpecName: "home") pod "4a203d7c-e245-4812-b7e8-3517c39f0954" (UID: "4a203d7c-e245-4812-b7e8-3517c39f0954"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:18:06.457036 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.456994    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a203d7c-e245-4812-b7e8-3517c39f0954" (UID: "4a203d7c-e245-4812-b7e8-3517c39f0954"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:18:06.458995 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.458965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-dshm" (OuterVolumeSpecName: "dshm") pod "4a203d7c-e245-4812-b7e8-3517c39f0954" (UID: "4a203d7c-e245-4812-b7e8-3517c39f0954"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:18:06.458995 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.458977    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a203d7c-e245-4812-b7e8-3517c39f0954-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a203d7c-e245-4812-b7e8-3517c39f0954" (UID: "4a203d7c-e245-4812-b7e8-3517c39f0954"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:18:06.458995 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.458989    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a203d7c-e245-4812-b7e8-3517c39f0954-kube-api-access-m62vv" (OuterVolumeSpecName: "kube-api-access-m62vv") pod "4a203d7c-e245-4812-b7e8-3517c39f0954" (UID: "4a203d7c-e245-4812-b7e8-3517c39f0954"). InnerVolumeSpecName "kube-api-access-m62vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:18:06.557567 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.557514    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:18:06.557567 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.557558    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m62vv\" (UniqueName: \"kubernetes.io/projected/4a203d7c-e245-4812-b7e8-3517c39f0954-kube-api-access-m62vv\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:18:06.557567 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.557571    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:18:06.557567 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.557580    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:18:06.557847 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.557610    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a203d7c-e245-4812-b7e8-3517c39f0954-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:18:06.557847 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:06.557618    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a203d7c-e245-4812-b7e8-3517c39f0954-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:18:07.390111 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:07.390079    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"
Sep 16 20:18:07.390587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:07.390083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8" event={"ID":"4a203d7c-e245-4812-b7e8-3517c39f0954","Type":"ContainerDied","Data":"115d40848465687fc02c172ba21d395de2a4ff85e982cf7bd5edd0237500effa"}
Sep 16 20:18:07.390587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:07.390214    2561 scope.go:117] "RemoveContainer" containerID="7326a68f849bd4d2d056ec9903c0a1416423c2477faedc68329b515928b80f01"
Sep 16 20:18:07.414560 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:07.414527    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"]
Sep 16 20:18:07.420991 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:07.420969    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-622k8"]
Sep 16 20:18:07.681907 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:07.681813    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a203d7c-e245-4812-b7e8-3517c39f0954" path="/var/lib/kubelet/pods/4a203d7c-e245-4812-b7e8-3517c39f0954/volumes"
Sep 16 20:18:46.491012 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.490978    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"]
Sep 16 20:18:46.491500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.491265    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" containerName="main"
Sep 16 20:18:46.491500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.491278    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" containerName="main"
Sep 16 20:18:46.491500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.491294    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a203d7c-e245-4812-b7e8-3517c39f0954" containerName="main"
Sep 16 20:18:46.491500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.491302    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a203d7c-e245-4812-b7e8-3517c39f0954" containerName="main"
Sep 16 20:18:46.491500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.491343    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e8da7a-5279-4bf4-8e27-a2a5f96c053a" containerName="main"
Sep 16 20:18:46.491500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.491350    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a203d7c-e245-4812-b7e8-3517c39f0954" containerName="main"
Sep 16 20:18:46.495438 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.495420    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.498420 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.498391    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 20:18:46.499771 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.499747    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 20:18:46.499860 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.499778    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 20:18:46.499860 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.499797    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-slm8b\""
Sep 16 20:18:46.505615 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.505591    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"]
Sep 16 20:18:46.563336 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.563301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-dshm\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.563336 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.563340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.563595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.563358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tls-certs\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.563595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.563394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-home\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.563595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.563436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxphl\" (UniqueName: \"kubernetes.io/projected/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-kube-api-access-vxphl\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.563595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.563575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-model-cache\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664147 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-model-cache\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664147 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-dshm\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664334 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664334 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tls-certs\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664334 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-home\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664471 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxphl\" (UniqueName: \"kubernetes.io/projected/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-kube-api-access-vxphl\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664639 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664705 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-model-cache\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.664767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.664745    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-home\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.666691 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.666665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-dshm\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.666899 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.666883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tls-certs\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.672704 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.672683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxphl\" (UniqueName: \"kubernetes.io/projected/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-kube-api-access-vxphl\") pod \"tri-v3-kserve-798c88c66c-7rzt2\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:46.805843 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:46.805803    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:47.085296 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:47.085218    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"]
Sep 16 20:18:47.088289 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:18:47.088263    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3955f5_4649_4b07_84ad_1efcdb1a2daa.slice/crio-6f9e0d0ecc697f2aecc0c32b448e2a155ae6283a406320903ecfb566f8154072 WatchSource:0}: Error finding container 6f9e0d0ecc697f2aecc0c32b448e2a155ae6283a406320903ecfb566f8154072: Status 404 returned error can't find the container with id 6f9e0d0ecc697f2aecc0c32b448e2a155ae6283a406320903ecfb566f8154072
Sep 16 20:18:47.548889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:47.548852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2" event={"ID":"0a3955f5-4649-4b07-84ad-1efcdb1a2daa","Type":"ContainerStarted","Data":"70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf"}
Sep 16 20:18:47.549307 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:47.548904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2" event={"ID":"0a3955f5-4649-4b07-84ad-1efcdb1a2daa","Type":"ContainerStarted","Data":"6f9e0d0ecc697f2aecc0c32b448e2a155ae6283a406320903ecfb566f8154072"}
Sep 16 20:18:47.569127 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:47.569082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2" podStartSLOduration=1.56906821 podStartE2EDuration="1.56906821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:18:47.568348939 +0000 UTC m=+2374.340219339" watchObservedRunningTime="2026-09-16 20:18:47.56906821 +0000 UTC m=+2374.340938620"
Sep 16 20:18:47.806352 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:47.806275    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:56.806674 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:56.806633    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:56.819217 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:56.819192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:18:57.601011 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:18:57.600980    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:19:13.772773 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:13.772744    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:19:13.778481 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:13.778461    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:19:30.555747 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.555708    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"]
Sep 16 20:19:30.558816 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.558796    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:30.561845 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.561823    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 20:19:30.563324 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.563296    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 20:19:30.563324 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.563316    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 20:19:30.563508 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.563318    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-7jzc9\""
Sep 16 20:19:30.563508 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.563428    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-lhn98\""
Sep 16 20:19:30.570051 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.570029    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"]
Sep 16 20:19:30.593959 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.593926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvxp\" (UniqueName: \"kubernetes.io/projected/d1a3260b-e3cd-44d1-9144-4818de1e35f5-kube-api-access-jpvxp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm\" (UID: \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:30.594086 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.593984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3260b-e3cd-44d1-9144-4818de1e35f5-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm\" (UID: \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:30.694862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.694824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3260b-e3cd-44d1-9144-4818de1e35f5-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm\" (UID: \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:30.695055 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.694924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvxp\" (UniqueName: \"kubernetes.io/projected/d1a3260b-e3cd-44d1-9144-4818de1e35f5-kube-api-access-jpvxp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm\" (UID: \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:30.697405 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.697366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3260b-e3cd-44d1-9144-4818de1e35f5-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm\" (UID: \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:30.704125 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.704100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvxp\" (UniqueName: \"kubernetes.io/projected/d1a3260b-e3cd-44d1-9144-4818de1e35f5-kube-api-access-jpvxp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm\" (UID: \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:30.868561 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:30.868480    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:31.146176 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:31.146101    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"]
Sep 16 20:19:31.149120 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:19:31.149091    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a3260b_e3cd_44d1_9144_4818de1e35f5.slice/crio-ce5a7982d4d5420e95b8a665142a1aa697cc6bba4cb7b5daa2e6b00b80b2fd5f WatchSource:0}: Error finding container ce5a7982d4d5420e95b8a665142a1aa697cc6bba4cb7b5daa2e6b00b80b2fd5f: Status 404 returned error can't find the container with id ce5a7982d4d5420e95b8a665142a1aa697cc6bba4cb7b5daa2e6b00b80b2fd5f
Sep 16 20:19:31.731337 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:31.731298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" event={"ID":"d1a3260b-e3cd-44d1-9144-4818de1e35f5","Type":"ContainerStarted","Data":"908d39ec0199255437084169e9bac851e9d9767fd633ae1dea5bfc2374eb61c4"}
Sep 16 20:19:31.731721 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:31.731359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" event={"ID":"d1a3260b-e3cd-44d1-9144-4818de1e35f5","Type":"ContainerStarted","Data":"ce5a7982d4d5420e95b8a665142a1aa697cc6bba4cb7b5daa2e6b00b80b2fd5f"}
Sep 16 20:19:31.731721 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:31.731489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:19:31.755659 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:19:31.755609    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" podStartSLOduration=1.755593239 podStartE2EDuration="1.755593239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:19:31.754139869 +0000 UTC m=+2418.526010282" watchObservedRunningTime="2026-09-16 20:19:31.755593239 +0000 UTC m=+2418.527463650"
Sep 16 20:20:02.737111 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:02.737079    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:20:28.454862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:28.454826    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"]
Sep 16 20:20:43.473685 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.473633    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2" podUID="0a3955f5-4649-4b07-84ad-1efcdb1a2daa" containerName="main" containerID="cri-o://70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf" gracePeriod=45
Sep 16 20:20:43.590601 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.590433    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2" podUID="0a3955f5-4649-4b07-84ad-1efcdb1a2daa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 20:20:43.723166 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.723137    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:20:43.872604 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872575    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-model-cache\") pod \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") "
Sep 16 20:20:43.872800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872612    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tmp-dir\") pod \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") "
Sep 16 20:20:43.872800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872660    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-home\") pod \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") "
Sep 16 20:20:43.872800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872680    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tls-certs\") pod \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") "
Sep 16 20:20:43.872800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872706    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-dshm\") pod \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") "
Sep 16 20:20:43.872800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872761    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxphl\" (UniqueName: \"kubernetes.io/projected/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-kube-api-access-vxphl\") pod \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\" (UID: \"0a3955f5-4649-4b07-84ad-1efcdb1a2daa\") "
Sep 16 20:20:43.873036 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872905    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a3955f5-4649-4b07-84ad-1efcdb1a2daa" (UID: "0a3955f5-4649-4b07-84ad-1efcdb1a2daa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.873036 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a3955f5-4649-4b07-84ad-1efcdb1a2daa" (UID: "0a3955f5-4649-4b07-84ad-1efcdb1a2daa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.873036 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.872973    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-home" (OuterVolumeSpecName: "home") pod "0a3955f5-4649-4b07-84ad-1efcdb1a2daa" (UID: "0a3955f5-4649-4b07-84ad-1efcdb1a2daa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.874825 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.874800    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a3955f5-4649-4b07-84ad-1efcdb1a2daa" (UID: "0a3955f5-4649-4b07-84ad-1efcdb1a2daa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:20:43.874938 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.874855    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-dshm" (OuterVolumeSpecName: "dshm") pod "0a3955f5-4649-4b07-84ad-1efcdb1a2daa" (UID: "0a3955f5-4649-4b07-84ad-1efcdb1a2daa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.874938 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.874865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-kube-api-access-vxphl" (OuterVolumeSpecName: "kube-api-access-vxphl") pod "0a3955f5-4649-4b07-84ad-1efcdb1a2daa" (UID: "0a3955f5-4649-4b07-84ad-1efcdb1a2daa"). InnerVolumeSpecName "kube-api-access-vxphl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:20:43.974180 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.974143    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxphl\" (UniqueName: \"kubernetes.io/projected/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-kube-api-access-vxphl\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.974180 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.974174    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.974180 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.974184    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.974180 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.974192    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.974480 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.974199    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.974480 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:43.974206    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3955f5-4649-4b07-84ad-1efcdb1a2daa-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:20:44.037154 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.037115    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a3955f5-4649-4b07-84ad-1efcdb1a2daa" containerID="70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf" exitCode=0
Sep 16 20:20:44.037314 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.037164    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"
Sep 16 20:20:44.037314 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.037196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2" event={"ID":"0a3955f5-4649-4b07-84ad-1efcdb1a2daa","Type":"ContainerDied","Data":"70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf"}
Sep 16 20:20:44.037314 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.037250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2" event={"ID":"0a3955f5-4649-4b07-84ad-1efcdb1a2daa","Type":"ContainerDied","Data":"6f9e0d0ecc697f2aecc0c32b448e2a155ae6283a406320903ecfb566f8154072"}
Sep 16 20:20:44.037314 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.037272    2561 scope.go:117] "RemoveContainer" containerID="70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf"
Sep 16 20:20:44.046258 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.046238    2561 scope.go:117] "RemoveContainer" containerID="70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf"
Sep 16 20:20:44.046559 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:20:44.046542    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf\": container with ID starting with 70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf not found: ID does not exist" containerID="70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf"
Sep 16 20:20:44.046609 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.046566    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf"} err="failed to get container status \"70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf\": rpc error: code = NotFound desc = could not find container \"70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf\": container with ID starting with 70aa109811ff39a881d9171842601a6dbd168f9488e7daeb00ea14c878bcfedf not found: ID does not exist"
Sep 16 20:20:44.057982 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.057959    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"]
Sep 16 20:20:44.061481 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:44.061461    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7rzt2"]
Sep 16 20:20:45.686691 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:20:45.686658    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3955f5-4649-4b07-84ad-1efcdb1a2daa" path="/var/lib/kubelet/pods/0a3955f5-4649-4b07-84ad-1efcdb1a2daa/volumes"
Sep 16 20:22:19.445369 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:22:19.445335    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"]
Sep 16 20:22:22.737281 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:22:22.737239    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:22:32.737807 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:22:32.737759    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:22:34.464427 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:22:34.464369    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main" containerID="cri-o://908d39ec0199255437084169e9bac851e9d9767fd633ae1dea5bfc2374eb61c4" gracePeriod=45
Sep 16 20:22:42.737215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:22:42.737128    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:22:42.737627 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:22:42.737247    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:22:52.737097 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:22:52.737053    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:23:02.737551 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:02.737511    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:23:04.609932 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.609900    2561 generic.go:356] "Generic (PLEG): container finished" podID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerID="908d39ec0199255437084169e9bac851e9d9767fd633ae1dea5bfc2374eb61c4" exitCode=0
Sep 16 20:23:04.610277 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.609980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" event={"ID":"d1a3260b-e3cd-44d1-9144-4818de1e35f5","Type":"ContainerDied","Data":"908d39ec0199255437084169e9bac851e9d9767fd633ae1dea5bfc2374eb61c4"}
Sep 16 20:23:04.706880 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.706856    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:23:04.819015 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.818982    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3260b-e3cd-44d1-9144-4818de1e35f5-tls-certs\") pod \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\" (UID: \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\") "
Sep 16 20:23:04.819200 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.819030    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpvxp\" (UniqueName: \"kubernetes.io/projected/d1a3260b-e3cd-44d1-9144-4818de1e35f5-kube-api-access-jpvxp\") pod \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\" (UID: \"d1a3260b-e3cd-44d1-9144-4818de1e35f5\") "
Sep 16 20:23:04.821293 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.821262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a3260b-e3cd-44d1-9144-4818de1e35f5-kube-api-access-jpvxp" (OuterVolumeSpecName: "kube-api-access-jpvxp") pod "d1a3260b-e3cd-44d1-9144-4818de1e35f5" (UID: "d1a3260b-e3cd-44d1-9144-4818de1e35f5"). InnerVolumeSpecName "kube-api-access-jpvxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:23:04.821293 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.821273    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a3260b-e3cd-44d1-9144-4818de1e35f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1a3260b-e3cd-44d1-9144-4818de1e35f5" (UID: "d1a3260b-e3cd-44d1-9144-4818de1e35f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:23:04.920541 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.920500    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3260b-e3cd-44d1-9144-4818de1e35f5-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:23:04.920541 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:04.920537    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvxp\" (UniqueName: \"kubernetes.io/projected/d1a3260b-e3cd-44d1-9144-4818de1e35f5-kube-api-access-jpvxp\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:23:05.615097 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:05.615061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm" event={"ID":"d1a3260b-e3cd-44d1-9144-4818de1e35f5","Type":"ContainerDied","Data":"ce5a7982d4d5420e95b8a665142a1aa697cc6bba4cb7b5daa2e6b00b80b2fd5f"}
Sep 16 20:23:05.615553 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:05.615118    2561 scope.go:117] "RemoveContainer" containerID="908d39ec0199255437084169e9bac851e9d9767fd633ae1dea5bfc2374eb61c4"
Sep 16 20:23:05.615553 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:05.615071    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"
Sep 16 20:23:05.638370 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:05.638345    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"]
Sep 16 20:23:05.644502 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:05.644479    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schen9xsm"]
Sep 16 20:23:05.682516 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:05.682491    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" path="/var/lib/kubelet/pods/d1a3260b-e3cd-44d1-9144-4818de1e35f5/volumes"
Sep 16 20:23:38.763778 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.763739    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"]
Sep 16 20:23:38.764167 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.764015    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3955f5-4649-4b07-84ad-1efcdb1a2daa" containerName="main"
Sep 16 20:23:38.764167 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.764026    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3955f5-4649-4b07-84ad-1efcdb1a2daa" containerName="main"
Sep 16 20:23:38.764167 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.764042    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main"
Sep 16 20:23:38.764167 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.764048    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main"
Sep 16 20:23:38.764167 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.764106    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3955f5-4649-4b07-84ad-1efcdb1a2daa" containerName="main"
Sep 16 20:23:38.764167 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.764115    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a3260b-e3cd-44d1-9144-4818de1e35f5" containerName="main"
Sep 16 20:23:38.768164 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.768138    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.772716 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.772679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-lklx4\""
Sep 16 20:23:38.772900 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.772679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 20:23:38.772900 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.772731    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 20:23:38.772900 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.772687    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 20:23:38.778283 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.778251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"]
Sep 16 20:23:38.877720 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.877682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19902c4-948a-407b-a6cd-05a9bf013a16-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.877720 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.877726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9v6\" (UniqueName: \"kubernetes.io/projected/a19902c4-948a-407b-a6cd-05a9bf013a16-kube-api-access-6p9v6\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.877958 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.877831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.877958 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.877865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.877958 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.877887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.877958 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.877909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-home\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979046 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979046 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979275 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979275 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-home\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979275 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19902c4-948a-407b-a6cd-05a9bf013a16-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979275 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9v6\" (UniqueName: \"kubernetes.io/projected/a19902c4-948a-407b-a6cd-05a9bf013a16-kube-api-access-6p9v6\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979521 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979598 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.979659 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.979634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-home\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.981423 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.981404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.981679 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.981661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19902c4-948a-407b-a6cd-05a9bf013a16-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:38.987850 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:38.987829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9v6\" (UniqueName: \"kubernetes.io/projected/a19902c4-948a-407b-a6cd-05a9bf013a16-kube-api-access-6p9v6\") pod \"delw-v2-kserve-5b84cc7ddf-dphw5\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:39.079788 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:39.079690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:39.356587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:39.356391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"]
Sep 16 20:23:39.359400 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:23:39.359356    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19902c4_948a_407b_a6cd_05a9bf013a16.slice/crio-b3bf01877704dcdfcd18d35a92ac0710451fdb1522b18a64f2bc9493b2cd23f1 WatchSource:0}: Error finding container b3bf01877704dcdfcd18d35a92ac0710451fdb1522b18a64f2bc9493b2cd23f1: Status 404 returned error can't find the container with id b3bf01877704dcdfcd18d35a92ac0710451fdb1522b18a64f2bc9493b2cd23f1
Sep 16 20:23:39.753389 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:39.753338    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5" event={"ID":"a19902c4-948a-407b-a6cd-05a9bf013a16","Type":"ContainerStarted","Data":"0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605"}
Sep 16 20:23:39.753560 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:39.753418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5" event={"ID":"a19902c4-948a-407b-a6cd-05a9bf013a16","Type":"ContainerStarted","Data":"b3bf01877704dcdfcd18d35a92ac0710451fdb1522b18a64f2bc9493b2cd23f1"}
Sep 16 20:23:39.813901 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:39.813846    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5" podStartSLOduration=1.813827563 podStartE2EDuration="1.813827563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:23:39.811771344 +0000 UTC m=+2666.583641755" watchObservedRunningTime="2026-09-16 20:23:39.813827563 +0000 UTC m=+2666.585697973"
Sep 16 20:23:40.080778 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:40.080665    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:49.080706 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:49.080662    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:49.094171 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:49.094134    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:23:49.805640 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:23:49.805607    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:24:13.796729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:13.796653    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:24:13.803918 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:13.803894    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:24:37.883050 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.883014    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"]
Sep 16 20:24:37.886099 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.886083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:37.889211 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.889189    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 20:24:37.889324 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.889188    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 20:24:37.890279 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.890255    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 20:24:37.890408 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.890353    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-gcz2m\""
Sep 16 20:24:37.897905 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.897883    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"]
Sep 16 20:24:37.923566 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.923535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:37.923683 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.923570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:37.923683 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.923591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:37.923758 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.923718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trc8t\" (UniqueName: \"kubernetes.io/projected/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-kube-api-access-trc8t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:37.923798 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.923754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:37.923798 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:37.923792    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025066 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trc8t\" (UniqueName: \"kubernetes.io/projected/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-kube-api-access-trc8t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025263 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025263 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025263 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025263 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025263 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025663 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025663 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.025663 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.025604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.027555 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.027533    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.027708 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.027693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.034521 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.034501    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trc8t\" (UniqueName: \"kubernetes.io/projected/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-kube-api-access-trc8t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.102355 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.102314    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"]
Sep 16 20:24:38.106142 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.106119    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:38.108921 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.108895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-jv5ls\""
Sep 16 20:24:38.115895 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.115868    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"]
Sep 16 20:24:38.196450 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.196353    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:38.227251 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.227216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2b6420-7e8b-4e68-b0f5-551fb591d500-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj\" (UID: \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:38.227415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.227333    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhjm\" (UniqueName: \"kubernetes.io/projected/3b2b6420-7e8b-4e68-b0f5-551fb591d500-kube-api-access-bvhjm\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj\" (UID: \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:38.328514 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.328480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhjm\" (UniqueName: \"kubernetes.io/projected/3b2b6420-7e8b-4e68-b0f5-551fb591d500-kube-api-access-bvhjm\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj\" (UID: \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:38.328700 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.328543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2b6420-7e8b-4e68-b0f5-551fb591d500-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj\" (UID: \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:38.330980 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.330959    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2b6420-7e8b-4e68-b0f5-551fb591d500-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj\" (UID: \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:38.336454 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.336425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhjm\" (UniqueName: \"kubernetes.io/projected/3b2b6420-7e8b-4e68-b0f5-551fb591d500-kube-api-access-bvhjm\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj\" (UID: \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:38.417446 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.417412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:38.467791 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.467763    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"]
Sep 16 20:24:38.471984 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:24:38.471956    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f8adf8_4fcc_4656_b4ce_ceddd18d6402.slice/crio-710e9d68a898fdff55a507d37c1b7a37af9180900f9467c9594a8a0fd8deb43c WatchSource:0}: Error finding container 710e9d68a898fdff55a507d37c1b7a37af9180900f9467c9594a8a0fd8deb43c: Status 404 returned error can't find the container with id 710e9d68a898fdff55a507d37c1b7a37af9180900f9467c9594a8a0fd8deb43c
Sep 16 20:24:38.688755 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.688585    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"]
Sep 16 20:24:38.691345 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:24:38.691308    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2b6420_7e8b_4e68_b0f5_551fb591d500.slice/crio-b66c09cf81e76027c1021d02ca7dc48752da58aca909d16fe4b2e4b4b11a1bf0 WatchSource:0}: Error finding container b66c09cf81e76027c1021d02ca7dc48752da58aca909d16fe4b2e4b4b11a1bf0: Status 404 returned error can't find the container with id b66c09cf81e76027c1021d02ca7dc48752da58aca909d16fe4b2e4b4b11a1bf0
Sep 16 20:24:38.992147 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.992043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk" event={"ID":"05f8adf8-4fcc-4656-b4ce-ceddd18d6402","Type":"ContainerStarted","Data":"7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1"}
Sep 16 20:24:38.992147 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.992112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk" event={"ID":"05f8adf8-4fcc-4656-b4ce-ceddd18d6402","Type":"ContainerStarted","Data":"710e9d68a898fdff55a507d37c1b7a37af9180900f9467c9594a8a0fd8deb43c"}
Sep 16 20:24:38.993658 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.993633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" event={"ID":"3b2b6420-7e8b-4e68-b0f5-551fb591d500","Type":"ContainerStarted","Data":"46906140d1fe24f5aecd8e5ef27f04cda2d5e0e5dec4a719d61c0c516fc6cda1"}
Sep 16 20:24:38.993787 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.993682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" event={"ID":"3b2b6420-7e8b-4e68-b0f5-551fb591d500","Type":"ContainerStarted","Data":"b66c09cf81e76027c1021d02ca7dc48752da58aca909d16fe4b2e4b4b11a1bf0"}
Sep 16 20:24:38.993787 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:38.993709    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:24:39.012426 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:39.012352    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk" podStartSLOduration=2.012334405 podStartE2EDuration="2.012334405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:24:39.01199 +0000 UTC m=+2725.783860437" watchObservedRunningTime="2026-09-16 20:24:39.012334405 +0000 UTC m=+2725.784204817"
Sep 16 20:24:39.029591 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:39.029550    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" podStartSLOduration=1.029536375 podStartE2EDuration="1.029536375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:24:39.02921878 +0000 UTC m=+2725.801089197" watchObservedRunningTime="2026-09-16 20:24:39.029536375 +0000 UTC m=+2725.801406785"
Sep 16 20:24:39.197410 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:39.197358    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:48.197535 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:48.197495    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:48.210163 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:48.210133    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:49.043348 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:49.043318    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:24:54.091477 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:24:54.091438    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"]
Sep 16 20:25:09.110287 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.110242    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5" podUID="a19902c4-948a-407b-a6cd-05a9bf013a16" containerName="main" containerID="cri-o://0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605" gracePeriod=45
Sep 16 20:25:09.356200 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.356172    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:25:09.480841 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.480765    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-tmp-dir\") pod \"a19902c4-948a-407b-a6cd-05a9bf013a16\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") "
Sep 16 20:25:09.480841 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.480824    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-model-cache\") pod \"a19902c4-948a-407b-a6cd-05a9bf013a16\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") "
Sep 16 20:25:09.481065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.480866    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-dshm\") pod \"a19902c4-948a-407b-a6cd-05a9bf013a16\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") "
Sep 16 20:25:09.481065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.480891    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19902c4-948a-407b-a6cd-05a9bf013a16-tls-certs\") pod \"a19902c4-948a-407b-a6cd-05a9bf013a16\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") "
Sep 16 20:25:09.481065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.480921    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p9v6\" (UniqueName: \"kubernetes.io/projected/a19902c4-948a-407b-a6cd-05a9bf013a16-kube-api-access-6p9v6\") pod \"a19902c4-948a-407b-a6cd-05a9bf013a16\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") "
Sep 16 20:25:09.481065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.480946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-home\") pod \"a19902c4-948a-407b-a6cd-05a9bf013a16\" (UID: \"a19902c4-948a-407b-a6cd-05a9bf013a16\") "
Sep 16 20:25:09.481065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.481029    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a19902c4-948a-407b-a6cd-05a9bf013a16" (UID: "a19902c4-948a-407b-a6cd-05a9bf013a16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:09.481335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.481070    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-model-cache" (OuterVolumeSpecName: "model-cache") pod "a19902c4-948a-407b-a6cd-05a9bf013a16" (UID: "a19902c4-948a-407b-a6cd-05a9bf013a16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:09.481335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.481221    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:09.481335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.481263    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:09.481335 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.481265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-home" (OuterVolumeSpecName: "home") pod "a19902c4-948a-407b-a6cd-05a9bf013a16" (UID: "a19902c4-948a-407b-a6cd-05a9bf013a16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:09.483257 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.483235    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-dshm" (OuterVolumeSpecName: "dshm") pod "a19902c4-948a-407b-a6cd-05a9bf013a16" (UID: "a19902c4-948a-407b-a6cd-05a9bf013a16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:09.483337 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.483246    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19902c4-948a-407b-a6cd-05a9bf013a16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a19902c4-948a-407b-a6cd-05a9bf013a16" (UID: "a19902c4-948a-407b-a6cd-05a9bf013a16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:25:09.483337 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.483256    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19902c4-948a-407b-a6cd-05a9bf013a16-kube-api-access-6p9v6" (OuterVolumeSpecName: "kube-api-access-6p9v6") pod "a19902c4-948a-407b-a6cd-05a9bf013a16" (UID: "a19902c4-948a-407b-a6cd-05a9bf013a16"). InnerVolumeSpecName "kube-api-access-6p9v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:25:09.582488 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.582454    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:09.582488 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.582482    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19902c4-948a-407b-a6cd-05a9bf013a16-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:09.582683 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.582498    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p9v6\" (UniqueName: \"kubernetes.io/projected/a19902c4-948a-407b-a6cd-05a9bf013a16-kube-api-access-6p9v6\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:09.582683 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.582507    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19902c4-948a-407b-a6cd-05a9bf013a16-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:09.998715 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:09.998687    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:25:10.116622 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.116583    2561 generic.go:356] "Generic (PLEG): container finished" podID="a19902c4-948a-407b-a6cd-05a9bf013a16" containerID="0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605" exitCode=0
Sep 16 20:25:10.117092 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.116629    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"
Sep 16 20:25:10.117092 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.116668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5" event={"ID":"a19902c4-948a-407b-a6cd-05a9bf013a16","Type":"ContainerDied","Data":"0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605"}
Sep 16 20:25:10.117092 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.116736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5" event={"ID":"a19902c4-948a-407b-a6cd-05a9bf013a16","Type":"ContainerDied","Data":"b3bf01877704dcdfcd18d35a92ac0710451fdb1522b18a64f2bc9493b2cd23f1"}
Sep 16 20:25:10.117092 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.116771    2561 scope.go:117] "RemoveContainer" containerID="0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605"
Sep 16 20:25:10.125026 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.124987    2561 scope.go:117] "RemoveContainer" containerID="0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605"
Sep 16 20:25:10.125349 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:25:10.125322    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605\": container with ID starting with 0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605 not found: ID does not exist" containerID="0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605"
Sep 16 20:25:10.125349 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.125352    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605"} err="failed to get container status \"0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605\": rpc error: code = NotFound desc = could not find container \"0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605\": container with ID starting with 0a46515627e5add412696c36a1c51775eb85bf10ff5d754d7278b5fc062ba605 not found: ID does not exist"
Sep 16 20:25:10.136789 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.136759    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"]
Sep 16 20:25:10.139208 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:10.139185    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-dphw5"]
Sep 16 20:25:11.682194 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:11.682157    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19902c4-948a-407b-a6cd-05a9bf013a16" path="/var/lib/kubelet/pods/a19902c4-948a-407b-a6cd-05a9bf013a16/volumes"
Sep 16 20:25:38.243288 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.243208    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"]
Sep 16 20:25:38.259109 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.259078    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"]
Sep 16 20:25:38.267788 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:25:38.267740    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:25:38.267788 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:25:38.267788 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:25:38.267788 ip-10-0-136-120 kubenswrapper[2561]:  > containerID="7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1" cmd=["/bin/sleep","15"]
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.267806    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk" message="" err=<
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]:  >
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:25:38.267819    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk" podUID="05f8adf8-4fcc-4656-b4ce-ceddd18d6402" containerName="main" containerID="cri-o://7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1"
Sep 16 20:25:38.268110 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.267867    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk" podUID="05f8adf8-4fcc-4656-b4ce-ceddd18d6402" containerName="main" containerID="cri-o://7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1" gracePeriod=60
Sep 16 20:25:38.614056 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.614031    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:25:38.695472 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695437    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-home\") pod \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") "
Sep 16 20:25:38.695692 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695533    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trc8t\" (UniqueName: \"kubernetes.io/projected/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-kube-api-access-trc8t\") pod \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") "
Sep 16 20:25:38.695692 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695560    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tmp-dir\") pod \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") "
Sep 16 20:25:38.695692 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695611    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tls-certs\") pod \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") "
Sep 16 20:25:38.695692 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-model-cache\") pod \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") "
Sep 16 20:25:38.695692 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695671    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-dshm\") pod \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\" (UID: \"05f8adf8-4fcc-4656-b4ce-ceddd18d6402\") "
Sep 16 20:25:38.695968 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695765    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-home" (OuterVolumeSpecName: "home") pod "05f8adf8-4fcc-4656-b4ce-ceddd18d6402" (UID: "05f8adf8-4fcc-4656-b4ce-ceddd18d6402"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:38.695968 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695921    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:38.695968 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.695956    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-model-cache" (OuterVolumeSpecName: "model-cache") pod "05f8adf8-4fcc-4656-b4ce-ceddd18d6402" (UID: "05f8adf8-4fcc-4656-b4ce-ceddd18d6402"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:38.696140 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.696105    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05f8adf8-4fcc-4656-b4ce-ceddd18d6402" (UID: "05f8adf8-4fcc-4656-b4ce-ceddd18d6402"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:38.697820 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.697790    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-dshm" (OuterVolumeSpecName: "dshm") pod "05f8adf8-4fcc-4656-b4ce-ceddd18d6402" (UID: "05f8adf8-4fcc-4656-b4ce-ceddd18d6402"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:38.697956 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.697843    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-kube-api-access-trc8t" (OuterVolumeSpecName: "kube-api-access-trc8t") pod "05f8adf8-4fcc-4656-b4ce-ceddd18d6402" (UID: "05f8adf8-4fcc-4656-b4ce-ceddd18d6402"). InnerVolumeSpecName "kube-api-access-trc8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:25:38.697956 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.697858    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05f8adf8-4fcc-4656-b4ce-ceddd18d6402" (UID: "05f8adf8-4fcc-4656-b4ce-ceddd18d6402"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:25:38.797249 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.797162    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trc8t\" (UniqueName: \"kubernetes.io/projected/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-kube-api-access-trc8t\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:38.797249 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.797193    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:38.797249 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.797204    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:38.797249 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.797212    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:38.797249 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:38.797220    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f8adf8-4fcc-4656-b4ce-ceddd18d6402-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:25:39.248690 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.248657    2561 generic.go:356] "Generic (PLEG): container finished" podID="05f8adf8-4fcc-4656-b4ce-ceddd18d6402" containerID="7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1" exitCode=0
Sep 16 20:25:39.249087 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.248709    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"
Sep 16 20:25:39.249087 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.248745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk" event={"ID":"05f8adf8-4fcc-4656-b4ce-ceddd18d6402","Type":"ContainerDied","Data":"7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1"}
Sep 16 20:25:39.249087 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.248807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk" event={"ID":"05f8adf8-4fcc-4656-b4ce-ceddd18d6402","Type":"ContainerDied","Data":"710e9d68a898fdff55a507d37c1b7a37af9180900f9467c9594a8a0fd8deb43c"}
Sep 16 20:25:39.249087 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.248834    2561 scope.go:117] "RemoveContainer" containerID="7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1"
Sep 16 20:25:39.257303 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.257280    2561 scope.go:117] "RemoveContainer" containerID="7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1"
Sep 16 20:25:39.257597 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:25:39.257574    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1\": container with ID starting with 7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1 not found: ID does not exist" containerID="7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1"
Sep 16 20:25:39.257702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.257603    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1"} err="failed to get container status \"7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1\": rpc error: code = NotFound desc = could not find container \"7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1\": container with ID starting with 7dac65f104490739fc7705bc7ca047b6c992fc9989e713938b44aee39fb13cd1 not found: ID does not exist"
Sep 16 20:25:39.270517 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.270488    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"]
Sep 16 20:25:39.276441 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.276420    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-762fk"]
Sep 16 20:25:39.682986 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.682903    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f8adf8-4fcc-4656-b4ce-ceddd18d6402" path="/var/lib/kubelet/pods/05f8adf8-4fcc-4656-b4ce-ceddd18d6402/volumes"
Sep 16 20:25:39.999155 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:39.999096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:25:49.998942 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:49.998903    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:25:53.262276 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:53.262238    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main" containerID="cri-o://46906140d1fe24f5aecd8e5ef27f04cda2d5e0e5dec4a719d61c0c516fc6cda1" gracePeriod=45
Sep 16 20:25:59.998757 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:59.998719    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:25:59.999249 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:25:59.998823    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:26:09.999058 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:09.999012    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:19.998573 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:19.998532    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:23.431984 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.431953    2561 generic.go:356] "Generic (PLEG): container finished" podID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerID="46906140d1fe24f5aecd8e5ef27f04cda2d5e0e5dec4a719d61c0c516fc6cda1" exitCode=0
Sep 16 20:26:23.432400 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.432012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" event={"ID":"3b2b6420-7e8b-4e68-b0f5-551fb591d500","Type":"ContainerDied","Data":"46906140d1fe24f5aecd8e5ef27f04cda2d5e0e5dec4a719d61c0c516fc6cda1"}
Sep 16 20:26:23.508145 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.508122    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:26:23.632238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.632147    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvhjm\" (UniqueName: \"kubernetes.io/projected/3b2b6420-7e8b-4e68-b0f5-551fb591d500-kube-api-access-bvhjm\") pod \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\" (UID: \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\") "
Sep 16 20:26:23.632238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.632200    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2b6420-7e8b-4e68-b0f5-551fb591d500-tls-certs\") pod \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\" (UID: \"3b2b6420-7e8b-4e68-b0f5-551fb591d500\") "
Sep 16 20:26:23.634297 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.634272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2b6420-7e8b-4e68-b0f5-551fb591d500-kube-api-access-bvhjm" (OuterVolumeSpecName: "kube-api-access-bvhjm") pod "3b2b6420-7e8b-4e68-b0f5-551fb591d500" (UID: "3b2b6420-7e8b-4e68-b0f5-551fb591d500"). InnerVolumeSpecName "kube-api-access-bvhjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:26:23.634402 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.634306    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2b6420-7e8b-4e68-b0f5-551fb591d500-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b2b6420-7e8b-4e68-b0f5-551fb591d500" (UID: "3b2b6420-7e8b-4e68-b0f5-551fb591d500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:26:23.733431 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.733397    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvhjm\" (UniqueName: \"kubernetes.io/projected/3b2b6420-7e8b-4e68-b0f5-551fb591d500-kube-api-access-bvhjm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:26:23.733431 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:23.733424    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2b6420-7e8b-4e68-b0f5-551fb591d500-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:26:24.437509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:24.437477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj" event={"ID":"3b2b6420-7e8b-4e68-b0f5-551fb591d500","Type":"ContainerDied","Data":"b66c09cf81e76027c1021d02ca7dc48752da58aca909d16fe4b2e4b4b11a1bf0"}
Sep 16 20:26:24.437509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:24.437502    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"
Sep 16 20:26:24.438083 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:24.437533    2561 scope.go:117] "RemoveContainer" containerID="46906140d1fe24f5aecd8e5ef27f04cda2d5e0e5dec4a719d61c0c516fc6cda1"
Sep 16 20:26:24.453616 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:24.453592    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"]
Sep 16 20:26:24.457780 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:24.457760    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schec77lj"]
Sep 16 20:26:25.682668 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:26:25.682625    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" path="/var/lib/kubelet/pods/3b2b6420-7e8b-4e68-b0f5-551fb591d500/volumes"
Sep 16 20:27:06.993004 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.992932    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"]
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993176    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19902c4-948a-407b-a6cd-05a9bf013a16" containerName="main"
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993186    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19902c4-948a-407b-a6cd-05a9bf013a16" containerName="main"
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993203    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f8adf8-4fcc-4656-b4ce-ceddd18d6402" containerName="main"
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993209    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f8adf8-4fcc-4656-b4ce-ceddd18d6402" containerName="main"
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993216    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main"
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993222    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main"
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993269    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f8adf8-4fcc-4656-b4ce-ceddd18d6402" containerName="main"
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993276    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19902c4-948a-407b-a6cd-05a9bf013a16" containerName="main"
Sep 16 20:27:06.993437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.993281    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2b6420-7e8b-4e68-b0f5-551fb591d500" containerName="main"
Sep 16 20:27:06.996080 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.996064    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:06.999102 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.999078    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 20:27:06.999213 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:06.999129    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-95rkl\""
Sep 16 20:27:07.000293 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.000272    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 20:27:07.000432 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.000273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 16 20:27:07.009470 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.009447    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"]
Sep 16 20:27:07.060435 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.060399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-dshm\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.060613 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.060442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-model-cache\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.060613 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.060496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.060613 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.060564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntw55\" (UniqueName: \"kubernetes.io/projected/71c97996-037f-41d9-b043-cbd7f6e5ebb7-kube-api-access-ntw55\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.060613 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.060583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-home\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.060749 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.060636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.160972 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.160931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-model-cache\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.160972 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.160970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.161191 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.161098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntw55\" (UniqueName: \"kubernetes.io/projected/71c97996-037f-41d9-b043-cbd7f6e5ebb7-kube-api-access-ntw55\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.161191 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.161137    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-home\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.161191 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.161162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.161347 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.161217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-dshm\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.161450 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.161419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-model-cache\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.161518 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.161481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.161576 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.161553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-home\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.163530 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.163508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-dshm\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.163701 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.163684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.169945 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.169921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntw55\" (UniqueName: \"kubernetes.io/projected/71c97996-037f-41d9-b043-cbd7f6e5ebb7-kube-api-access-ntw55\") pod \"owner-v1-kserve-75579ff5bc-8cpbq\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.305880 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.305842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:07.582835 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.582811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"]
Sep 16 20:27:07.585447 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:27:07.585415    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c97996_037f_41d9_b043_cbd7f6e5ebb7.slice/crio-20f6f83d1e1f254af32442e786fd41039574dca1c2851b4d68f785d39d25ae1f WatchSource:0}: Error finding container 20f6f83d1e1f254af32442e786fd41039574dca1c2851b4d68f785d39d25ae1f: Status 404 returned error can't find the container with id 20f6f83d1e1f254af32442e786fd41039574dca1c2851b4d68f785d39d25ae1f
Sep 16 20:27:07.610311 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:07.610284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq" event={"ID":"71c97996-037f-41d9-b043-cbd7f6e5ebb7","Type":"ContainerStarted","Data":"20f6f83d1e1f254af32442e786fd41039574dca1c2851b4d68f785d39d25ae1f"}
Sep 16 20:27:08.615767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:08.615733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq" event={"ID":"71c97996-037f-41d9-b043-cbd7f6e5ebb7","Type":"ContainerStarted","Data":"c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5"}
Sep 16 20:27:08.638232 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:08.638178    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq" podStartSLOduration=2.638161013 podStartE2EDuration="2.638161013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:08.637370136 +0000 UTC m=+2875.409240546" watchObservedRunningTime="2026-09-16 20:27:08.638161013 +0000 UTC m=+2875.410031425"
Sep 16 20:27:09.306767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:09.306726    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:17.306666 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:17.306626    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:17.318922 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:17.318892    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:17.661410 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:17.661310    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:27:40.596918 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.596881    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"]
Sep 16 20:27:40.600272 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.600247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.603500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.603471    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 20:27:40.603650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.603471    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 20:27:40.603650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.603598    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 20:27:40.603650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.603627    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-r85t4\""
Sep 16 20:27:40.603806 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.603475    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 20:27:40.603806 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.603747    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:27:40.615259 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.615235    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"]
Sep 16 20:27:40.715560 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.715529    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpd2m\" (UniqueName: \"kubernetes.io/projected/025f4129-abdf-45cc-b117-ededdbcddb13-kube-api-access-zpd2m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.715744 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.715570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/025f4129-abdf-45cc-b117-ededdbcddb13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.715744 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.715589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.715744 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.715612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.715744 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.715650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.715744 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.715711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025f4129-abdf-45cc-b117-ededdbcddb13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.715744 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.715735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.715951 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.715752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816291 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025f4129-abdf-45cc-b117-ededdbcddb13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816525 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816525 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816525 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpd2m\" (UniqueName: \"kubernetes.io/projected/025f4129-abdf-45cc-b117-ededdbcddb13-kube-api-access-zpd2m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816726 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/025f4129-abdf-45cc-b117-ededdbcddb13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816726 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816726 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816726 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.816983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.816958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.817045 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.817006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.817137 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.817115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.817197 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.817159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.817329 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.817312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/025f4129-abdf-45cc-b117-ededdbcddb13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.818717 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.818701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.819128 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.819107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025f4129-abdf-45cc-b117-ededdbcddb13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.828601 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.828571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpd2m\" (UniqueName: \"kubernetes.io/projected/025f4129-abdf-45cc-b117-ededdbcddb13-kube-api-access-zpd2m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:40.910764 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:40.910686    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:41.187928 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:41.187844    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"]
Sep 16 20:27:41.190800 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:27:41.190764    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025f4129_abdf_45cc_b117_ededdbcddb13.slice/crio-e054e4c3ec25e2f1a74416e940b0def4af813acd839f004755360fb12207b58b WatchSource:0}: Error finding container e054e4c3ec25e2f1a74416e940b0def4af813acd839f004755360fb12207b58b: Status 404 returned error can't find the container with id e054e4c3ec25e2f1a74416e940b0def4af813acd839f004755360fb12207b58b
Sep 16 20:27:41.747397 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:41.747346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" event={"ID":"025f4129-abdf-45cc-b117-ededdbcddb13","Type":"ContainerStarted","Data":"78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74"}
Sep 16 20:27:41.747783 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:41.747421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" event={"ID":"025f4129-abdf-45cc-b117-ededdbcddb13","Type":"ContainerStarted","Data":"e054e4c3ec25e2f1a74416e940b0def4af813acd839f004755360fb12207b58b"}
Sep 16 20:27:44.763342 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:44.763307    2561 generic.go:356] "Generic (PLEG): container finished" podID="025f4129-abdf-45cc-b117-ededdbcddb13" containerID="78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74" exitCode=0
Sep 16 20:27:44.763708 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:44.763397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" event={"ID":"025f4129-abdf-45cc-b117-ededdbcddb13","Type":"ContainerDied","Data":"78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74"}
Sep 16 20:27:45.772843 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:45.772807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" event={"ID":"025f4129-abdf-45cc-b117-ededdbcddb13","Type":"ContainerStarted","Data":"eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775"}
Sep 16 20:27:45.797811 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:45.797755    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podStartSLOduration=3.143156164 podStartE2EDuration="5.797738724s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.65458256s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:45.79633898 +0000 UTC m=+2912.568209391" watchObservedRunningTime="2026-09-16 20:27:45.797738724 +0000 UTC m=+2912.569609135"
Sep 16 20:27:45.911718 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:45.911684    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:50.911075 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:50.911042    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:27:50.912584 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:50.912549    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 20:27:58.395863 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:27:58.395823    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"]
Sep 16 20:28:00.911888 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:00.911854    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 20:28:10.911919 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:10.911875    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 20:28:13.416016 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.415941    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq" podUID="71c97996-037f-41d9-b043-cbd7f6e5ebb7" containerName="main" containerID="cri-o://c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5" gracePeriod=45
Sep 16 20:28:13.662616 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.662590    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:28:13.688897 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.688825    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-model-cache\") pod \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") "
Sep 16 20:28:13.688897 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.688873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-dshm\") pod \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") "
Sep 16 20:28:13.689102 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.688951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-home\") pod \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") "
Sep 16 20:28:13.689102 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.688999    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tmp-dir\") pod \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") "
Sep 16 20:28:13.689102 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.689024    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tls-certs\") pod \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") "
Sep 16 20:28:13.689102 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.689054    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntw55\" (UniqueName: \"kubernetes.io/projected/71c97996-037f-41d9-b043-cbd7f6e5ebb7-kube-api-access-ntw55\") pod \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\" (UID: \"71c97996-037f-41d9-b043-cbd7f6e5ebb7\") "
Sep 16 20:28:13.689273 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.689143    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "71c97996-037f-41d9-b043-cbd7f6e5ebb7" (UID: "71c97996-037f-41d9-b043-cbd7f6e5ebb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:28:13.689333 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.689310    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:28:13.689630 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.689603    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71c97996-037f-41d9-b043-cbd7f6e5ebb7" (UID: "71c97996-037f-41d9-b043-cbd7f6e5ebb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:28:13.689761 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.689736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-home" (OuterVolumeSpecName: "home") pod "71c97996-037f-41d9-b043-cbd7f6e5ebb7" (UID: "71c97996-037f-41d9-b043-cbd7f6e5ebb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:28:13.691679 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.691650    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-dshm" (OuterVolumeSpecName: "dshm") pod "71c97996-037f-41d9-b043-cbd7f6e5ebb7" (UID: "71c97996-037f-41d9-b043-cbd7f6e5ebb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:28:13.691849 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.691826    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71c97996-037f-41d9-b043-cbd7f6e5ebb7" (UID: "71c97996-037f-41d9-b043-cbd7f6e5ebb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:28:13.691849 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.691839    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c97996-037f-41d9-b043-cbd7f6e5ebb7-kube-api-access-ntw55" (OuterVolumeSpecName: "kube-api-access-ntw55") pod "71c97996-037f-41d9-b043-cbd7f6e5ebb7" (UID: "71c97996-037f-41d9-b043-cbd7f6e5ebb7"). InnerVolumeSpecName "kube-api-access-ntw55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:28:13.789934 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.789897    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:28:13.789934 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.789926    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:28:13.789934 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.789936    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c97996-037f-41d9-b043-cbd7f6e5ebb7-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:28:13.790171 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.789946    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntw55\" (UniqueName: \"kubernetes.io/projected/71c97996-037f-41d9-b043-cbd7f6e5ebb7-kube-api-access-ntw55\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:28:13.790171 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.789954    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c97996-037f-41d9-b043-cbd7f6e5ebb7-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:28:13.900729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.900688    2561 generic.go:356] "Generic (PLEG): container finished" podID="71c97996-037f-41d9-b043-cbd7f6e5ebb7" containerID="c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5" exitCode=0
Sep 16 20:28:13.900924 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.900735    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"
Sep 16 20:28:13.900924 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.900749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq" event={"ID":"71c97996-037f-41d9-b043-cbd7f6e5ebb7","Type":"ContainerDied","Data":"c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5"}
Sep 16 20:28:13.900924 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.900796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq" event={"ID":"71c97996-037f-41d9-b043-cbd7f6e5ebb7","Type":"ContainerDied","Data":"20f6f83d1e1f254af32442e786fd41039574dca1c2851b4d68f785d39d25ae1f"}
Sep 16 20:28:13.900924 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.900821    2561 scope.go:117] "RemoveContainer" containerID="c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5"
Sep 16 20:28:13.909912 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.909772    2561 scope.go:117] "RemoveContainer" containerID="c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5"
Sep 16 20:28:13.910178 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:28:13.910148    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5\": container with ID starting with c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5 not found: ID does not exist" containerID="c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5"
Sep 16 20:28:13.910266 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.910192    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5"} err="failed to get container status \"c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5\": rpc error: code = NotFound desc = could not find container \"c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5\": container with ID starting with c93044ce2ee3e16026bfd39aaa6c114293a994d52ae813b626b4757340160db5 not found: ID does not exist"
Sep 16 20:28:13.924203 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.924180    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"]
Sep 16 20:28:13.929325 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:13.929301    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq"]
Sep 16 20:28:14.651328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:14.651234    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-8cpbq" podUID="71c97996-037f-41d9-b043-cbd7f6e5ebb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded"
Sep 16 20:28:15.682063 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:15.682018    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c97996-037f-41d9-b043-cbd7f6e5ebb7" path="/var/lib/kubelet/pods/71c97996-037f-41d9-b043-cbd7f6e5ebb7/volumes"
Sep 16 20:28:20.911749 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:20.911706    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 20:28:30.911336 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:30.911288    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 20:28:40.911778 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:40.911688    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 20:28:50.911597 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:28:50.911556    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 20:29:00.921804 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:00.921770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:29:00.929759 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:00.929729    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:29:13.821896 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:13.821862    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:29:13.831121 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:13.831096    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:29:26.510320 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:26.510283    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"]
Sep 16 20:29:41.532019 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:41.531944    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" containerID="cri-o://eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775" gracePeriod=45
Sep 16 20:29:41.923162 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:41.923059    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 20:29:42.181906 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.181848    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:29:42.265430 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.265364    2561 generic.go:356] "Generic (PLEG): container finished" podID="025f4129-abdf-45cc-b117-ededdbcddb13" containerID="eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775" exitCode=0
Sep 16 20:29:42.265430 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.265411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" event={"ID":"025f4129-abdf-45cc-b117-ededdbcddb13","Type":"ContainerDied","Data":"eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775"}
Sep 16 20:29:42.265624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.265445    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"
Sep 16 20:29:42.265624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.265473    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf" event={"ID":"025f4129-abdf-45cc-b117-ededdbcddb13","Type":"ContainerDied","Data":"e054e4c3ec25e2f1a74416e940b0def4af813acd839f004755360fb12207b58b"}
Sep 16 20:29:42.265624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.265496    2561 scope.go:117] "RemoveContainer" containerID="eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775"
Sep 16 20:29:42.273673 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.273655    2561 scope.go:117] "RemoveContainer" containerID="78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74"
Sep 16 20:29:42.281219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.281203    2561 scope.go:117] "RemoveContainer" containerID="eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775"
Sep 16 20:29:42.281508 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:29:42.281488    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775\": container with ID starting with eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775 not found: ID does not exist" containerID="eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775"
Sep 16 20:29:42.281559 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.281516    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775"} err="failed to get container status \"eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775\": rpc error: code = NotFound desc = could not find container \"eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775\": container with ID starting with eed20378b9c93a1fb59038460885a68022f8144143e8eeda071f12a411db8775 not found: ID does not exist"
Sep 16 20:29:42.281559 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.281534    2561 scope.go:117] "RemoveContainer" containerID="78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74"
Sep 16 20:29:42.281786 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:29:42.281764    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74\": container with ID starting with 78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74 not found: ID does not exist" containerID="78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74"
Sep 16 20:29:42.281833 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.281793    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74"} err="failed to get container status \"78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74\": rpc error: code = NotFound desc = could not find container \"78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74\": container with ID starting with 78cb2182a2c4d16b15cc9a683b8b4a83f65aff78dd0002a9ba185daeae742f74 not found: ID does not exist"
Sep 16 20:29:42.293029 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/025f4129-abdf-45cc-b117-ededdbcddb13-cabundle-cert\") pod \"025f4129-abdf-45cc-b117-ededdbcddb13\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") "
Sep 16 20:29:42.293121 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293047    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-dshm\") pod \"025f4129-abdf-45cc-b117-ededdbcddb13\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") "
Sep 16 20:29:42.293121 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-tmp-dir\") pod \"025f4129-abdf-45cc-b117-ededdbcddb13\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") "
Sep 16 20:29:42.293248 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293233    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-model-cache\") pod \"025f4129-abdf-45cc-b117-ededdbcddb13\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") "
Sep 16 20:29:42.293306 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293276    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025f4129-abdf-45cc-b117-ededdbcddb13-tls-certs\") pod \"025f4129-abdf-45cc-b117-ededdbcddb13\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") "
Sep 16 20:29:42.293369 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293311    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-kserve-provision-location\") pod \"025f4129-abdf-45cc-b117-ededdbcddb13\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") "
Sep 16 20:29:42.293369 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293346    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-home\") pod \"025f4129-abdf-45cc-b117-ededdbcddb13\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") "
Sep 16 20:29:42.293493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293420    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpd2m\" (UniqueName: \"kubernetes.io/projected/025f4129-abdf-45cc-b117-ededdbcddb13-kube-api-access-zpd2m\") pod \"025f4129-abdf-45cc-b117-ededdbcddb13\" (UID: \"025f4129-abdf-45cc-b117-ededdbcddb13\") "
Sep 16 20:29:42.293493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025f4129-abdf-45cc-b117-ededdbcddb13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "025f4129-abdf-45cc-b117-ededdbcddb13" (UID: "025f4129-abdf-45cc-b117-ededdbcddb13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:29:42.293729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293683    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/025f4129-abdf-45cc-b117-ededdbcddb13-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:29:42.293729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293523    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-model-cache" (OuterVolumeSpecName: "model-cache") pod "025f4129-abdf-45cc-b117-ededdbcddb13" (UID: "025f4129-abdf-45cc-b117-ededdbcddb13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:42.293904 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.293878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-home" (OuterVolumeSpecName: "home") pod "025f4129-abdf-45cc-b117-ededdbcddb13" (UID: "025f4129-abdf-45cc-b117-ededdbcddb13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:42.295907 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.295871    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025f4129-abdf-45cc-b117-ededdbcddb13-kube-api-access-zpd2m" (OuterVolumeSpecName: "kube-api-access-zpd2m") pod "025f4129-abdf-45cc-b117-ededdbcddb13" (UID: "025f4129-abdf-45cc-b117-ededdbcddb13"). InnerVolumeSpecName "kube-api-access-zpd2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:29:42.295907 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.295877    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-dshm" (OuterVolumeSpecName: "dshm") pod "025f4129-abdf-45cc-b117-ededdbcddb13" (UID: "025f4129-abdf-45cc-b117-ededdbcddb13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:42.302638 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.302343    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025f4129-abdf-45cc-b117-ededdbcddb13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "025f4129-abdf-45cc-b117-ededdbcddb13" (UID: "025f4129-abdf-45cc-b117-ededdbcddb13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:29:42.307976 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.307953    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "025f4129-abdf-45cc-b117-ededdbcddb13" (UID: "025f4129-abdf-45cc-b117-ededdbcddb13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:42.321781 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.321749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "025f4129-abdf-45cc-b117-ededdbcddb13" (UID: "025f4129-abdf-45cc-b117-ededdbcddb13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:42.395071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.395038    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:29:42.395071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.395068    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:29:42.395071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.395077    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025f4129-abdf-45cc-b117-ededdbcddb13-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:29:42.395071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.395085    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:29:42.395328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.395093    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:29:42.395328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.395102    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpd2m\" (UniqueName: \"kubernetes.io/projected/025f4129-abdf-45cc-b117-ededdbcddb13-kube-api-access-zpd2m\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:29:42.395328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.395110    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025f4129-abdf-45cc-b117-ededdbcddb13-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:29:42.588533 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.588479    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"]
Sep 16 20:29:42.592784 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:42.592757    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-69458cffbd-w4wqf"]
Sep 16 20:29:43.682915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:29:43.682876    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" path="/var/lib/kubelet/pods/025f4129-abdf-45cc-b117-ededdbcddb13/volumes"
Sep 16 20:30:03.215461 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215422    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"]
Sep 16 20:30:03.215856 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215680    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="storage-initializer"
Sep 16 20:30:03.215856 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215690    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="storage-initializer"
Sep 16 20:30:03.215856 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215703    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main"
Sep 16 20:30:03.215856 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215708    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main"
Sep 16 20:30:03.215856 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215716    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c97996-037f-41d9-b043-cbd7f6e5ebb7" containerName="main"
Sep 16 20:30:03.215856 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215722    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c97996-037f-41d9-b043-cbd7f6e5ebb7" containerName="main"
Sep 16 20:30:03.215856 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215768    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c97996-037f-41d9-b043-cbd7f6e5ebb7" containerName="main"
Sep 16 20:30:03.215856 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.215776    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="025f4129-abdf-45cc-b117-ededdbcddb13" containerName="main"
Sep 16 20:30:03.218354 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.218333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.221556 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.221533    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 20:30:03.221814 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.221792    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-kwv82\""
Sep 16 20:30:03.221975 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.221950    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 20:30:03.222055 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.221863    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:30:03.222140 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.222121    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 20:30:03.222644 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.222623    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 20:30:03.233455 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.233427    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"]
Sep 16 20:30:03.356578 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.356538    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a723d52c-0458-4e76-bad8-6a1c47434eae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.356783 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.356592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6qh\" (UniqueName: \"kubernetes.io/projected/a723d52c-0458-4e76-bad8-6a1c47434eae-kube-api-access-5n6qh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.356783 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.356626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.356783 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.356651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.356783 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.356674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.356783 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.356739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.356783 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.356772    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723d52c-0458-4e76-bad8-6a1c47434eae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.357037 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.356790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458076 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6qh\" (UniqueName: \"kubernetes.io/projected/a723d52c-0458-4e76-bad8-6a1c47434eae-kube-api-access-5n6qh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458076 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458341 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458341 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458341 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458341 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723d52c-0458-4e76-bad8-6a1c47434eae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458341 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458341 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a723d52c-0458-4e76-bad8-6a1c47434eae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458721 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458786 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458859 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.458976 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.458946    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.459307 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.459285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723d52c-0458-4e76-bad8-6a1c47434eae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.460588 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.460560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.461006 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.460990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a723d52c-0458-4e76-bad8-6a1c47434eae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.467293 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.467238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6qh\" (UniqueName: \"kubernetes.io/projected/a723d52c-0458-4e76-bad8-6a1c47434eae-kube-api-access-5n6qh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.491650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.491611    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"]
Sep 16 20:30:03.495872 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.495845    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:03.498868 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.498845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-dwkh7\""
Sep 16 20:30:03.506095 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.506070    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"]
Sep 16 20:30:03.529337 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.529301    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:03.660303 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.660256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz\" (UID: \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:03.660509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.660365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npp9z\" (UniqueName: \"kubernetes.io/projected/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-kube-api-access-npp9z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz\" (UID: \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:03.761427 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.761367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npp9z\" (UniqueName: \"kubernetes.io/projected/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-kube-api-access-npp9z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz\" (UID: \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:03.761631 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.761465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz\" (UID: \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:03.764218 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.764181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz\" (UID: \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:03.771658 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.771631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npp9z\" (UniqueName: \"kubernetes.io/projected/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-kube-api-access-npp9z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz\" (UID: \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:03.807631 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.807594    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:03.811662 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:03.811635    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"]
Sep 16 20:30:03.814809 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:30:03.814766    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda723d52c_0458_4e76_bad8_6a1c47434eae.slice/crio-a8739e4ecdac5b2519ee1de1140fa5c4296d5edbdda97d85903ca5469287dda4 WatchSource:0}: Error finding container a8739e4ecdac5b2519ee1de1140fa5c4296d5edbdda97d85903ca5469287dda4: Status 404 returned error can't find the container with id a8739e4ecdac5b2519ee1de1140fa5c4296d5edbdda97d85903ca5469287dda4
Sep 16 20:30:04.084874 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:04.084786    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"]
Sep 16 20:30:04.088002 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:30:04.087969    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a81fd0_b25b_4929_b1d4_c5aa779b5ff0.slice/crio-225c0afad2ff14f6618dbe1795a9355310f4e8df0e4225d6e13462f7f269a79d WatchSource:0}: Error finding container 225c0afad2ff14f6618dbe1795a9355310f4e8df0e4225d6e13462f7f269a79d: Status 404 returned error can't find the container with id 225c0afad2ff14f6618dbe1795a9355310f4e8df0e4225d6e13462f7f269a79d
Sep 16 20:30:04.355407 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:04.355285    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" event={"ID":"a723d52c-0458-4e76-bad8-6a1c47434eae","Type":"ContainerStarted","Data":"b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430"}
Sep 16 20:30:04.355407 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:04.355356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" event={"ID":"a723d52c-0458-4e76-bad8-6a1c47434eae","Type":"ContainerStarted","Data":"a8739e4ecdac5b2519ee1de1140fa5c4296d5edbdda97d85903ca5469287dda4"}
Sep 16 20:30:04.356873 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:04.356839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" event={"ID":"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0","Type":"ContainerStarted","Data":"fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5"}
Sep 16 20:30:04.356990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:04.356883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" event={"ID":"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0","Type":"ContainerStarted","Data":"225c0afad2ff14f6618dbe1795a9355310f4e8df0e4225d6e13462f7f269a79d"}
Sep 16 20:30:04.356990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:04.356977    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:04.403440 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:04.403387    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" podStartSLOduration=1.403350477 podStartE2EDuration="1.403350477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:30:04.402269868 +0000 UTC m=+3051.174140279" watchObservedRunningTime="2026-09-16 20:30:04.403350477 +0000 UTC m=+3051.175220889"
Sep 16 20:30:06.367146 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:06.367113    2561 generic.go:356] "Generic (PLEG): container finished" podID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerID="b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430" exitCode=0
Sep 16 20:30:06.367550 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:06.367190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" event={"ID":"a723d52c-0458-4e76-bad8-6a1c47434eae","Type":"ContainerDied","Data":"b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430"}
Sep 16 20:30:07.373097 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:07.373063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" event={"ID":"a723d52c-0458-4e76-bad8-6a1c47434eae","Type":"ContainerStarted","Data":"9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf"}
Sep 16 20:30:07.398425 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:07.398352    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podStartSLOduration=2.132403195 podStartE2EDuration="4.398333936s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.265930741s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:30:07.396008203 +0000 UTC m=+3054.167878638" watchObservedRunningTime="2026-09-16 20:30:07.398333936 +0000 UTC m=+3054.170204346"
Sep 16 20:30:07.529446 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:07.529406    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:13.529666 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:13.529623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:30:13.531211 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:13.531178    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:30:23.530390 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:23.530313    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:30:33.530074 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:33.530034    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:30:35.362690 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:35.362661    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:30:43.529870 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:43.529814    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:30:53.530731 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:30:53.530678    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:31:03.530443 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:31:03.530369    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:31:13.530394 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:31:13.530336    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:31:23.529868 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:31:23.529820    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:31:33.530368 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:31:33.530325    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:31:43.530065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:31:43.529966    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:31:53.530129 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:31:53.530082    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:32:03.539730 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:03.539694    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:32:03.549103 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:03.549063    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:32:09.026387 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:09.026337    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"]
Sep 16 20:32:09.041722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:09.041688    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"]
Sep 16 20:32:15.362752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:15.362707    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:32:24.048797 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.048735    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" containerID="cri-o://9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf" gracePeriod=45
Sep 16 20:32:24.059904 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.059867    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerName="main" containerID="cri-o://fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5" gracePeriod=45
Sep 16 20:32:24.541343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.541285    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 20:32:24.786357 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.786327    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:32:24.810993 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.810912    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-dshm\") pod \"a723d52c-0458-4e76-bad8-6a1c47434eae\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") "
Sep 16 20:32:24.810993 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.810961    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723d52c-0458-4e76-bad8-6a1c47434eae-cabundle-cert\") pod \"a723d52c-0458-4e76-bad8-6a1c47434eae\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") "
Sep 16 20:32:24.811205 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.811006    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a723d52c-0458-4e76-bad8-6a1c47434eae-tls-certs\") pod \"a723d52c-0458-4e76-bad8-6a1c47434eae\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") "
Sep 16 20:32:24.811205 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.811067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-tmp-dir\") pod \"a723d52c-0458-4e76-bad8-6a1c47434eae\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") "
Sep 16 20:32:24.811205 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.811138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-model-cache\") pod \"a723d52c-0458-4e76-bad8-6a1c47434eae\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") "
Sep 16 20:32:24.811205 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.811174    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-kserve-provision-location\") pod \"a723d52c-0458-4e76-bad8-6a1c47434eae\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") "
Sep 16 20:32:24.811500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.811216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n6qh\" (UniqueName: \"kubernetes.io/projected/a723d52c-0458-4e76-bad8-6a1c47434eae-kube-api-access-5n6qh\") pod \"a723d52c-0458-4e76-bad8-6a1c47434eae\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") "
Sep 16 20:32:24.811500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.811242    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-home\") pod \"a723d52c-0458-4e76-bad8-6a1c47434eae\" (UID: \"a723d52c-0458-4e76-bad8-6a1c47434eae\") "
Sep 16 20:32:24.812071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.812030    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-model-cache" (OuterVolumeSpecName: "model-cache") pod "a723d52c-0458-4e76-bad8-6a1c47434eae" (UID: "a723d52c-0458-4e76-bad8-6a1c47434eae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:24.812409 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.812358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-home" (OuterVolumeSpecName: "home") pod "a723d52c-0458-4e76-bad8-6a1c47434eae" (UID: "a723d52c-0458-4e76-bad8-6a1c47434eae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:24.814188 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.813744    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a723d52c-0458-4e76-bad8-6a1c47434eae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a723d52c-0458-4e76-bad8-6a1c47434eae" (UID: "a723d52c-0458-4e76-bad8-6a1c47434eae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:32:24.814188 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.814025    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-dshm" (OuterVolumeSpecName: "dshm") pod "a723d52c-0458-4e76-bad8-6a1c47434eae" (UID: "a723d52c-0458-4e76-bad8-6a1c47434eae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:24.814188 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.814087    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a723d52c-0458-4e76-bad8-6a1c47434eae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a723d52c-0458-4e76-bad8-6a1c47434eae" (UID: "a723d52c-0458-4e76-bad8-6a1c47434eae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:32:24.814544 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.814516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a723d52c-0458-4e76-bad8-6a1c47434eae-kube-api-access-5n6qh" (OuterVolumeSpecName: "kube-api-access-5n6qh") pod "a723d52c-0458-4e76-bad8-6a1c47434eae" (UID: "a723d52c-0458-4e76-bad8-6a1c47434eae"). InnerVolumeSpecName "kube-api-access-5n6qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:32:24.831242 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.831211    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a723d52c-0458-4e76-bad8-6a1c47434eae" (UID: "a723d52c-0458-4e76-bad8-6a1c47434eae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:24.840985 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.840962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a723d52c-0458-4e76-bad8-6a1c47434eae" (UID: "a723d52c-0458-4e76-bad8-6a1c47434eae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:24.912189 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.912155    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:24.912189 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.912187    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:24.912189 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.912198    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n6qh\" (UniqueName: \"kubernetes.io/projected/a723d52c-0458-4e76-bad8-6a1c47434eae-kube-api-access-5n6qh\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:24.912436 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.912208    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:24.912436 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.912216    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:24.912436 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.912224    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723d52c-0458-4e76-bad8-6a1c47434eae-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:24.912436 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.912233    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a723d52c-0458-4e76-bad8-6a1c47434eae-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:24.912436 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.912240    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a723d52c-0458-4e76-bad8-6a1c47434eae-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:24.966108 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.966074    2561 generic.go:356] "Generic (PLEG): container finished" podID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerID="9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf" exitCode=0
Sep 16 20:32:24.966260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.966125    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"
Sep 16 20:32:24.966260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.966163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" event={"ID":"a723d52c-0458-4e76-bad8-6a1c47434eae","Type":"ContainerDied","Data":"9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf"}
Sep 16 20:32:24.966260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.966228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4" event={"ID":"a723d52c-0458-4e76-bad8-6a1c47434eae","Type":"ContainerDied","Data":"a8739e4ecdac5b2519ee1de1140fa5c4296d5edbdda97d85903ca5469287dda4"}
Sep 16 20:32:24.966260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.966251    2561 scope.go:117] "RemoveContainer" containerID="9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf"
Sep 16 20:32:24.975138 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.975115    2561 scope.go:117] "RemoveContainer" containerID="b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430"
Sep 16 20:32:24.982619 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.982602    2561 scope.go:117] "RemoveContainer" containerID="9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf"
Sep 16 20:32:24.982905 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:32:24.982890    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf\": container with ID starting with 9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf not found: ID does not exist" containerID="9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf"
Sep 16 20:32:24.982956 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.982912    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf"} err="failed to get container status \"9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf\": rpc error: code = NotFound desc = could not find container \"9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf\": container with ID starting with 9f7209ff64947140c93fb5dc14b38003a1d339de0aea3f10c1ffae0fe885a1bf not found: ID does not exist"
Sep 16 20:32:24.982956 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.982928    2561 scope.go:117] "RemoveContainer" containerID="b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430"
Sep 16 20:32:24.983161 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:32:24.983142    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430\": container with ID starting with b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430 not found: ID does not exist" containerID="b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430"
Sep 16 20:32:24.983205 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.983168    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430"} err="failed to get container status \"b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430\": rpc error: code = NotFound desc = could not find container \"b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430\": container with ID starting with b005c6cce5ebc15db963258bec71afdf1e8280f0f06a3d00f0b643dcfe9f5430 not found: ID does not exist"
Sep 16 20:32:24.988997 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.988974    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"]
Sep 16 20:32:24.994915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:24.994892    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6fc9459b44sztd4"]
Sep 16 20:32:25.362804 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:25.362763    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:32:25.682969 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:25.682888    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" path="/var/lib/kubelet/pods/a723d52c-0458-4e76-bad8-6a1c47434eae/volumes"
Sep 16 20:32:35.362808 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:35.362767    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:32:35.363393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:35.362905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:32:45.362627 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:45.362587    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:32:54.300081 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:54.300051    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:32:54.442220 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:54.442129    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npp9z\" (UniqueName: \"kubernetes.io/projected/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-kube-api-access-npp9z\") pod \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\" (UID: \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\") "
Sep 16 20:32:54.442220 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:54.442177    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-tls-certs\") pod \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\" (UID: \"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0\") "
Sep 16 20:32:54.444365 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:54.444331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-kube-api-access-npp9z" (OuterVolumeSpecName: "kube-api-access-npp9z") pod "61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" (UID: "61a81fd0-b25b-4929-b1d4-c5aa779b5ff0"). InnerVolumeSpecName "kube-api-access-npp9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:32:54.444365 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:54.444343    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" (UID: "61a81fd0-b25b-4929-b1d4-c5aa779b5ff0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:32:54.543537 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:54.543504    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:54.543537 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:54.543532    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npp9z\" (UniqueName: \"kubernetes.io/projected/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0-kube-api-access-npp9z\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:32:55.082310 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.082268    2561 generic.go:356] "Generic (PLEG): container finished" podID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerID="fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5" exitCode=0
Sep 16 20:32:55.082310 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.082316    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"
Sep 16 20:32:55.082534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.082352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" event={"ID":"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0","Type":"ContainerDied","Data":"fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5"}
Sep 16 20:32:55.082534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.082418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz" event={"ID":"61a81fd0-b25b-4929-b1d4-c5aa779b5ff0","Type":"ContainerDied","Data":"225c0afad2ff14f6618dbe1795a9355310f4e8df0e4225d6e13462f7f269a79d"}
Sep 16 20:32:55.082534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.082441    2561 scope.go:117] "RemoveContainer" containerID="fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5"
Sep 16 20:32:55.090681 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.090661    2561 scope.go:117] "RemoveContainer" containerID="fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5"
Sep 16 20:32:55.090949 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:32:55.090927    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5\": container with ID starting with fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5 not found: ID does not exist" containerID="fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5"
Sep 16 20:32:55.091058 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.090959    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5"} err="failed to get container status \"fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5\": rpc error: code = NotFound desc = could not find container \"fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5\": container with ID starting with fd3926b177afe27ff5bff7bc696ef2a7f1f4fd6a3e68b42806e814384a10f9f5 not found: ID does not exist"
Sep 16 20:32:55.103640 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.103618    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"]
Sep 16 20:32:55.111385 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.111351    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef75tz"]
Sep 16 20:32:55.681910 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:32:55.681865    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" path="/var/lib/kubelet/pods/61a81fd0-b25b-4929-b1d4-c5aa779b5ff0/volumes"
Sep 16 20:33:01.742495 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742464    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"]
Sep 16 20:33:01.742862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742713    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="storage-initializer"
Sep 16 20:33:01.742862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742723    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="storage-initializer"
Sep 16 20:33:01.742862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742734    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main"
Sep 16 20:33:01.742862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742740    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main"
Sep 16 20:33:01.742862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742747    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerName="main"
Sep 16 20:33:01.742862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742752    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerName="main"
Sep 16 20:33:01.742862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742795    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a81fd0-b25b-4929-b1d4-c5aa779b5ff0" containerName="main"
Sep 16 20:33:01.742862 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.742801    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a723d52c-0458-4e76-bad8-6a1c47434eae" containerName="main"
Sep 16 20:33:01.745484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.745466    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.749737 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.749714    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-czptp\""
Sep 16 20:33:01.750160 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.750145    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 20:33:01.750469 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.750452    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 20:33:01.750738 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.750719    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 20:33:01.752042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.752025    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 20:33:01.779184 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.779158    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"]
Sep 16 20:33:01.898574 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.898535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.898767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.898574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.898767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.898637    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-dshm\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.898767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.898672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.898767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.898695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfmbh\" (UniqueName: \"kubernetes.io/projected/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kube-api-access-sfmbh\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.898767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.898731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-model-cache\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.898933 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.898789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-home\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.999900 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.999868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-dshm\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:01.999900 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.999902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000131 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.999924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfmbh\" (UniqueName: \"kubernetes.io/projected/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kube-api-access-sfmbh\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000131 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:01.999972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-model-cache\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000131 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.000012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-home\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000131 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.000113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.000167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.000322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000480 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.000406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-home\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000542 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.000494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-model-cache\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.000542 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.000517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.002209 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.002193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-dshm\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.002710 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.002685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.010262 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.010229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfmbh\" (UniqueName: \"kubernetes.io/projected/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kube-api-access-sfmbh\") pod \"e2e-del-blocked-svc-kserve-5d678b578b-7b5mm\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.056265 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.056227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:02.341490 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:02.341459    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"]
Sep 16 20:33:02.343712 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:33:02.343687    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7766264_43ea_4b8f_84b3_4d7caf8b59df.slice/crio-bd0518fad3508e91f6f88cded2df366386e4b89384330de132bbea9764144965 WatchSource:0}: Error finding container bd0518fad3508e91f6f88cded2df366386e4b89384330de132bbea9764144965: Status 404 returned error can't find the container with id bd0518fad3508e91f6f88cded2df366386e4b89384330de132bbea9764144965
Sep 16 20:33:03.122337 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:03.122290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm" event={"ID":"a7766264-43ea-4b8f-84b3-4d7caf8b59df","Type":"ContainerStarted","Data":"fb848f79ca1e2b6182df0cc3675892beae94857f732931cbc6deed43eab3ca4e"}
Sep 16 20:33:03.122821 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:03.122420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm" event={"ID":"a7766264-43ea-4b8f-84b3-4d7caf8b59df","Type":"ContainerStarted","Data":"bd0518fad3508e91f6f88cded2df366386e4b89384330de132bbea9764144965"}
Sep 16 20:33:04.539845 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:04.539809    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"]
Sep 16 20:33:05.131219 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:05.131157    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm" podUID="a7766264-43ea-4b8f-84b3-4d7caf8b59df" containerName="storage-initializer" containerID="cri-o://fb848f79ca1e2b6182df0cc3675892beae94857f732931cbc6deed43eab3ca4e" gracePeriod=60
Sep 16 20:33:07.141267 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.141234    2561 generic.go:356] "Generic (PLEG): container finished" podID="a7766264-43ea-4b8f-84b3-4d7caf8b59df" containerID="fb848f79ca1e2b6182df0cc3675892beae94857f732931cbc6deed43eab3ca4e" exitCode=0
Sep 16 20:33:07.141768 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.141288    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm" event={"ID":"a7766264-43ea-4b8f-84b3-4d7caf8b59df","Type":"ContainerDied","Data":"fb848f79ca1e2b6182df0cc3675892beae94857f732931cbc6deed43eab3ca4e"}
Sep 16 20:33:07.185071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.185042    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:07.344985 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.344892    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kserve-provision-location\") pod \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") "
Sep 16 20:33:07.344985 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.344943    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tls-certs\") pod \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") "
Sep 16 20:33:07.345222 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.344987    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tmp-dir\") pod \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") "
Sep 16 20:33:07.345222 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.345021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-dshm\") pod \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") "
Sep 16 20:33:07.345222 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.345043    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-home\") pod \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") "
Sep 16 20:33:07.345222 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.345138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfmbh\" (UniqueName: \"kubernetes.io/projected/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kube-api-access-sfmbh\") pod \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") "
Sep 16 20:33:07.345222 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.345181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-model-cache\") pod \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\" (UID: \"a7766264-43ea-4b8f-84b3-4d7caf8b59df\") "
Sep 16 20:33:07.345509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.345298    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7766264-43ea-4b8f-84b3-4d7caf8b59df" (UID: "a7766264-43ea-4b8f-84b3-4d7caf8b59df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:33:07.345509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.345485    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:33:07.345628 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.345510    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7766264-43ea-4b8f-84b3-4d7caf8b59df" (UID: "a7766264-43ea-4b8f-84b3-4d7caf8b59df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:33:07.345628 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.345621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-home" (OuterVolumeSpecName: "home") pod "a7766264-43ea-4b8f-84b3-4d7caf8b59df" (UID: "a7766264-43ea-4b8f-84b3-4d7caf8b59df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:33:07.347281 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.347255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-dshm" (OuterVolumeSpecName: "dshm") pod "a7766264-43ea-4b8f-84b3-4d7caf8b59df" (UID: "a7766264-43ea-4b8f-84b3-4d7caf8b59df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:33:07.347395 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.347338    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kube-api-access-sfmbh" (OuterVolumeSpecName: "kube-api-access-sfmbh") pod "a7766264-43ea-4b8f-84b3-4d7caf8b59df" (UID: "a7766264-43ea-4b8f-84b3-4d7caf8b59df"). InnerVolumeSpecName "kube-api-access-sfmbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:33:07.347470 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.347454    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7766264-43ea-4b8f-84b3-4d7caf8b59df" (UID: "a7766264-43ea-4b8f-84b3-4d7caf8b59df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:33:07.422218 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.422165    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7766264-43ea-4b8f-84b3-4d7caf8b59df" (UID: "a7766264-43ea-4b8f-84b3-4d7caf8b59df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:33:07.446789 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.446759    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfmbh\" (UniqueName: \"kubernetes.io/projected/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kube-api-access-sfmbh\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:33:07.446789 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.446787    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:33:07.446926 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.446796    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:33:07.446926 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.446807    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7766264-43ea-4b8f-84b3-4d7caf8b59df-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:33:07.446926 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.446815    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:33:07.446926 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:07.446823    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7766264-43ea-4b8f-84b3-4d7caf8b59df-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:33:08.146513 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:08.146479    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"
Sep 16 20:33:08.146914 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:08.146504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm" event={"ID":"a7766264-43ea-4b8f-84b3-4d7caf8b59df","Type":"ContainerDied","Data":"bd0518fad3508e91f6f88cded2df366386e4b89384330de132bbea9764144965"}
Sep 16 20:33:08.146914 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:08.146572    2561 scope.go:117] "RemoveContainer" containerID="fb848f79ca1e2b6182df0cc3675892beae94857f732931cbc6deed43eab3ca4e"
Sep 16 20:33:08.186604 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:08.186572    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"]
Sep 16 20:33:08.190694 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:08.190664    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5d678b578b-7b5mm"]
Sep 16 20:33:09.682262 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:09.682215    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7766264-43ea-4b8f-84b3-4d7caf8b59df" path="/var/lib/kubelet/pods/a7766264-43ea-4b8f-84b3-4d7caf8b59df/volumes"
Sep 16 20:33:27.897905 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.897862    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"]
Sep 16 20:33:27.898366 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.898124    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7766264-43ea-4b8f-84b3-4d7caf8b59df" containerName="storage-initializer"
Sep 16 20:33:27.898366 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.898136    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7766264-43ea-4b8f-84b3-4d7caf8b59df" containerName="storage-initializer"
Sep 16 20:33:27.898366 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.898186    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7766264-43ea-4b8f-84b3-4d7caf8b59df" containerName="storage-initializer"
Sep 16 20:33:27.901245 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.901219    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:27.904720 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.904663    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 20:33:27.904720 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.904663    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-6ptpw\""
Sep 16 20:33:27.904720 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.904663    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 20:33:27.904961 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.904673    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 20:33:27.915624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.915597    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"]
Sep 16 20:33:27.990251 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.990216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:27.990457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.990256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj74z\" (UniqueName: \"kubernetes.io/projected/8a5c8fa9-3ff4-4885-b577-a36015523c23-kube-api-access-mj74z\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:27.990457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.990279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:27.990457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.990332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:27.990457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.990365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5c8fa9-3ff4-4885-b577-a36015523c23-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:27.990457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.990419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:27.990457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:27.990436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091250 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091250 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5c8fa9-3ff4-4885-b577-a36015523c23-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091543 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091651 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj74z\" (UniqueName: \"kubernetes.io/projected/8a5c8fa9-3ff4-4885-b577-a36015523c23-kube-api-access-mj74z\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091710 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091765    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.091984 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.092057 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.091977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.093655 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.093633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.093930 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.093912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5c8fa9-3ff4-4885-b577-a36015523c23-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.105150 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.105120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj74z\" (UniqueName: \"kubernetes.io/projected/8a5c8fa9-3ff4-4885-b577-a36015523c23-kube-api-access-mj74z\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.194542 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.194458    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"]
Sep 16 20:33:28.198865 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.198848    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:28.201488 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.201466    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-m9nbf\""
Sep 16 20:33:28.207501 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.207468    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"]
Sep 16 20:33:28.212167 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.212145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:28.293636 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.293601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b925a2-4df6-41db-9cc6-5144e02c2aca-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p\" (UID: \"61b925a2-4df6-41db-9cc6-5144e02c2aca\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:28.293795 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.293658    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhjp\" (UniqueName: \"kubernetes.io/projected/61b925a2-4df6-41db-9cc6-5144e02c2aca-kube-api-access-8hhjp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p\" (UID: \"61b925a2-4df6-41db-9cc6-5144e02c2aca\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:28.394038 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.393995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b925a2-4df6-41db-9cc6-5144e02c2aca-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p\" (UID: \"61b925a2-4df6-41db-9cc6-5144e02c2aca\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:28.394038 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.394037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhjp\" (UniqueName: \"kubernetes.io/projected/61b925a2-4df6-41db-9cc6-5144e02c2aca-kube-api-access-8hhjp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p\" (UID: \"61b925a2-4df6-41db-9cc6-5144e02c2aca\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:28.397035 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.397010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b925a2-4df6-41db-9cc6-5144e02c2aca-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p\" (UID: \"61b925a2-4df6-41db-9cc6-5144e02c2aca\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:28.402622 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.402589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhjp\" (UniqueName: \"kubernetes.io/projected/61b925a2-4df6-41db-9cc6-5144e02c2aca-kube-api-access-8hhjp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p\" (UID: \"61b925a2-4df6-41db-9cc6-5144e02c2aca\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:28.491066 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.491037    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"]
Sep 16 20:33:28.510231 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.510193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:28.527587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.527569    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:33:28.784792 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:28.784764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"]
Sep 16 20:33:28.787034 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:33:28.787006    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b925a2_4df6_41db_9cc6_5144e02c2aca.slice/crio-ff862a8bd64b79b50a66ad0e20d2b500443247c4265ab3c1fac711362c5dfcb9 WatchSource:0}: Error finding container ff862a8bd64b79b50a66ad0e20d2b500443247c4265ab3c1fac711362c5dfcb9: Status 404 returned error can't find the container with id ff862a8bd64b79b50a66ad0e20d2b500443247c4265ab3c1fac711362c5dfcb9
Sep 16 20:33:29.244587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:29.244485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" event={"ID":"61b925a2-4df6-41db-9cc6-5144e02c2aca","Type":"ContainerStarted","Data":"86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460"}
Sep 16 20:33:29.244587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:29.244538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" event={"ID":"61b925a2-4df6-41db-9cc6-5144e02c2aca","Type":"ContainerStarted","Data":"ff862a8bd64b79b50a66ad0e20d2b500443247c4265ab3c1fac711362c5dfcb9"}
Sep 16 20:33:29.245060 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:29.244687    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:33:29.245829 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:29.245801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" event={"ID":"8a5c8fa9-3ff4-4885-b577-a36015523c23","Type":"ContainerStarted","Data":"0752a23ba5ac5e046e7857f488dee7dae61ffa4ea4fe70642d9f7bfa06aeace8"}
Sep 16 20:33:29.265006 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:29.264957    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" podStartSLOduration=1.2649388959999999 podStartE2EDuration="1.264938896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:33:29.263004315 +0000 UTC m=+3256.034874744" watchObservedRunningTime="2026-09-16 20:33:29.264938896 +0000 UTC m=+3256.036809309"
Sep 16 20:33:36.277954 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:36.277916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" event={"ID":"8a5c8fa9-3ff4-4885-b577-a36015523c23","Type":"ContainerStarted","Data":"77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69"}
Sep 16 20:33:38.287155 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:38.287121    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerID="77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69" exitCode=0
Sep 16 20:33:38.287619 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:38.287194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" event={"ID":"8a5c8fa9-3ff4-4885-b577-a36015523c23","Type":"ContainerDied","Data":"77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69"}
Sep 16 20:33:39.293672 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:39.293629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" event={"ID":"8a5c8fa9-3ff4-4885-b577-a36015523c23","Type":"ContainerStarted","Data":"0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727"}
Sep 16 20:33:39.294039 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:39.293723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" event={"ID":"8a5c8fa9-3ff4-4885-b577-a36015523c23","Type":"ContainerStarted","Data":"6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9"}
Sep 16 20:33:39.318727 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:39.318669    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podStartSLOduration=3.562576573 podStartE2EDuration="12.318654139s" totalImagesPullingTime="7.669831339s" totalInitContainerRuntime="1.086246227s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:33:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:33:39.317115997 +0000 UTC m=+3266.088986424" watchObservedRunningTime="2026-09-16 20:33:39.318654139 +0000 UTC m=+3266.090524550"
Sep 16 20:33:40.212742 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:40.212701    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:48.212362 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:48.212317    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:33:48.213790 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:48.213760    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:33:58.213257 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:33:58.213213    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:34:00.254146 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:00.254116    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:34:08.213220 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:08.213166    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:34:13.850438 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:13.850401    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:34:13.859273 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:13.859250    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:34:18.213281 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:18.213221    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:34:28.213021 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:28.212973    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:34:38.212823 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:38.212721    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:34:48.213199 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:48.213150    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:34:58.212959 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:34:58.212912    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:35:08.213465 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:08.213417    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:35:18.212779 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:18.212734    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:35:28.213534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:28.213487    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:35:38.222797 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:38.222764    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:35:38.230763 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:38.230737    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:35:43.358607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:43.358572    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"]
Sep 16 20:35:43.386764 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:43.386735    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"]
Sep 16 20:35:43.386997 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:43.386964    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="modelcar" containerID="cri-o://0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727" gracePeriod=60
Sep 16 20:35:43.841660 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:43.841625    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm_8a5c8fa9-3ff4-4885-b577-a36015523c23/modelcar/0.log"
Sep 16 20:35:43.842371 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:43.842339    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerID="0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727" exitCode=143
Sep 16 20:35:43.842520 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:43.842426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" event={"ID":"8a5c8fa9-3ff4-4885-b577-a36015523c23","Type":"ContainerDied","Data":"0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727"}
Sep 16 20:35:50.254417 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:50.254362    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:35:58.377249 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:58.377203    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerName="main" containerID="cri-o://86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460" gracePeriod=45
Sep 16 20:35:58.409183 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:58.409144    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" containerID="cri-o://6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9" gracePeriod=45
Sep 16 20:35:59.224238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.224155    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 20:35:59.652911 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.652885    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm_8a5c8fa9-3ff4-4885-b577-a36015523c23/modelcar/0.log"
Sep 16 20:35:59.655726 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.655703    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:35:59.707888 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.707857    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-dshm\") pod \"8a5c8fa9-3ff4-4885-b577-a36015523c23\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") "
Sep 16 20:35:59.708044 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.707893    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-model-cache\") pod \"8a5c8fa9-3ff4-4885-b577-a36015523c23\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") "
Sep 16 20:35:59.708044 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.707919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-home\") pod \"8a5c8fa9-3ff4-4885-b577-a36015523c23\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") "
Sep 16 20:35:59.708044 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.707957    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-kserve-provision-location\") pod \"8a5c8fa9-3ff4-4885-b577-a36015523c23\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") "
Sep 16 20:35:59.708044 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.708000    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5c8fa9-3ff4-4885-b577-a36015523c23-tls-certs\") pod \"8a5c8fa9-3ff4-4885-b577-a36015523c23\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") "
Sep 16 20:35:59.708044 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.708037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-tmp-dir\") pod \"8a5c8fa9-3ff4-4885-b577-a36015523c23\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") "
Sep 16 20:35:59.708329 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.708071    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj74z\" (UniqueName: \"kubernetes.io/projected/8a5c8fa9-3ff4-4885-b577-a36015523c23-kube-api-access-mj74z\") pod \"8a5c8fa9-3ff4-4885-b577-a36015523c23\" (UID: \"8a5c8fa9-3ff4-4885-b577-a36015523c23\") "
Sep 16 20:35:59.708329 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.708161    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a5c8fa9-3ff4-4885-b577-a36015523c23" (UID: "8a5c8fa9-3ff4-4885-b577-a36015523c23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:35:59.708329 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.708305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a5c8fa9-3ff4-4885-b577-a36015523c23" (UID: "8a5c8fa9-3ff4-4885-b577-a36015523c23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:35:59.708507 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.708370    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:35:59.708507 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.708407    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:35:59.708621 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.708565    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-home" (OuterVolumeSpecName: "home") pod "8a5c8fa9-3ff4-4885-b577-a36015523c23" (UID: "8a5c8fa9-3ff4-4885-b577-a36015523c23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:35:59.710217 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.710187    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-dshm" (OuterVolumeSpecName: "dshm") pod "8a5c8fa9-3ff4-4885-b577-a36015523c23" (UID: "8a5c8fa9-3ff4-4885-b577-a36015523c23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:35:59.710339 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.710328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5c8fa9-3ff4-4885-b577-a36015523c23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a5c8fa9-3ff4-4885-b577-a36015523c23" (UID: "8a5c8fa9-3ff4-4885-b577-a36015523c23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:35:59.710470 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.710450    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5c8fa9-3ff4-4885-b577-a36015523c23-kube-api-access-mj74z" (OuterVolumeSpecName: "kube-api-access-mj74z") pod "8a5c8fa9-3ff4-4885-b577-a36015523c23" (UID: "8a5c8fa9-3ff4-4885-b577-a36015523c23"). InnerVolumeSpecName "kube-api-access-mj74z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:35:59.720861 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.720835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a5c8fa9-3ff4-4885-b577-a36015523c23" (UID: "8a5c8fa9-3ff4-4885-b577-a36015523c23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:35:59.808955 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.808878    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj74z\" (UniqueName: \"kubernetes.io/projected/8a5c8fa9-3ff4-4885-b577-a36015523c23-kube-api-access-mj74z\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:35:59.808955 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.808906    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:35:59.808955 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.808917    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:35:59.808955 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.808926    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5c8fa9-3ff4-4885-b577-a36015523c23-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:35:59.808955 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.808934    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5c8fa9-3ff4-4885-b577-a36015523c23-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:35:59.908309 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.908283    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm_8a5c8fa9-3ff4-4885-b577-a36015523c23/modelcar/0.log"
Sep 16 20:35:59.909069 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.909044    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerID="6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9" exitCode=0
Sep 16 20:35:59.909137 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.909118    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"
Sep 16 20:35:59.909231 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.909117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" event={"ID":"8a5c8fa9-3ff4-4885-b577-a36015523c23","Type":"ContainerDied","Data":"6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9"}
Sep 16 20:35:59.909288 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.909273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm" event={"ID":"8a5c8fa9-3ff4-4885-b577-a36015523c23","Type":"ContainerDied","Data":"0752a23ba5ac5e046e7857f488dee7dae61ffa4ea4fe70642d9f7bfa06aeace8"}
Sep 16 20:35:59.909326 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.909307    2561 scope.go:117] "RemoveContainer" containerID="0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727"
Sep 16 20:35:59.926910 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.926891    2561 scope.go:117] "RemoveContainer" containerID="6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9"
Sep 16 20:35:59.934308 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.934289    2561 scope.go:117] "RemoveContainer" containerID="77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69"
Sep 16 20:35:59.937822 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.937800    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"]
Sep 16 20:35:59.942031 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.942008    2561 scope.go:117] "RemoveContainer" containerID="0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727"
Sep 16 20:35:59.942412 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:35:59.942330    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727\": container with ID starting with 0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727 not found: ID does not exist" containerID="0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727"
Sep 16 20:35:59.942412 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.942370    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727"} err="failed to get container status \"0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727\": rpc error: code = NotFound desc = could not find container \"0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727\": container with ID starting with 0ee5adc544327edd3c13f9eba315182a0023ccaabe01efcfe73c0084b65b6727 not found: ID does not exist"
Sep 16 20:35:59.942634 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.942442    2561 scope.go:117] "RemoveContainer" containerID="6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9"
Sep 16 20:35:59.942849 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:35:59.942743    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9\": container with ID starting with 6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9 not found: ID does not exist" containerID="6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9"
Sep 16 20:35:59.942849 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.942773    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9"} err="failed to get container status \"6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9\": rpc error: code = NotFound desc = could not find container \"6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9\": container with ID starting with 6ad3357bf9e4701663f0841391587f9ba205bd9a84ce4aaf86ce6456d462e9a9 not found: ID does not exist"
Sep 16 20:35:59.942849 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.942794    2561 scope.go:117] "RemoveContainer" containerID="77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69"
Sep 16 20:35:59.943107 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:35:59.943086    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69\": container with ID starting with 77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69 not found: ID does not exist" containerID="77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69"
Sep 16 20:35:59.943152 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.943120    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69"} err="failed to get container status \"77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69\": rpc error: code = NotFound desc = could not find container \"77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69\": container with ID starting with 77f5100249f74a54ee322fdf180c2a00be83a91f3a372e1c0ec60376d91b9e69 not found: ID does not exist"
Sep 16 20:35:59.944186 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:35:59.944165    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7qf4qm"]
Sep 16 20:36:00.254679 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:00.254641    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:36:01.682796 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:01.682764    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" path="/var/lib/kubelet/pods/8a5c8fa9-3ff4-4885-b577-a36015523c23/volumes"
Sep 16 20:36:06.421990 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.421916    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"]
Sep 16 20:36:06.422328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.422189    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main"
Sep 16 20:36:06.422328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.422200    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main"
Sep 16 20:36:06.422328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.422213    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="modelcar-init"
Sep 16 20:36:06.422328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.422219    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="modelcar-init"
Sep 16 20:36:06.422328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.422236    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="modelcar"
Sep 16 20:36:06.422328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.422244    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="modelcar"
Sep 16 20:36:06.422328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.422293    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="modelcar"
Sep 16 20:36:06.422328 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.422300    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5c8fa9-3ff4-4885-b577-a36015523c23" containerName="main"
Sep 16 20:36:06.426906 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.426883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.430722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.430705    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:36:06.430997 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.430980    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-pk8rp\""
Sep 16 20:36:06.431062 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.431045    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 20:36:06.431223 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.431206    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 20:36:06.431298 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.431283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 20:36:06.431341 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.431320    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 20:36:06.440258 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.440236    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"]
Sep 16 20:36:06.455081 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.455058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-kserve-provision-location\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.455199 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.455096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34534fe9-e4f9-406f-8f86-6826b20e35bc-tls-certs\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.455199 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.455114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-home\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.455199 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.455197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b75rs\" (UniqueName: \"kubernetes.io/projected/34534fe9-e4f9-406f-8f86-6826b20e35bc-kube-api-access-b75rs\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.455327 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.455216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-tmp-dir\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.455327 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.455242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34534fe9-e4f9-406f-8f86-6826b20e35bc-cabundle-cert\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.455327 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.455265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-dshm\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.455327 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.455285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-model-cache\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.555970 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.555930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-model-cache\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.555977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-kserve-provision-location\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34534fe9-e4f9-406f-8f86-6826b20e35bc-tls-certs\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-home\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b75rs\" (UniqueName: \"kubernetes.io/projected/34534fe9-e4f9-406f-8f86-6826b20e35bc-kube-api-access-b75rs\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-tmp-dir\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34534fe9-e4f9-406f-8f86-6826b20e35bc-cabundle-cert\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-dshm\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556586 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-model-cache\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556586 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-kserve-provision-location\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556586 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-home\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556730 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-tmp-dir\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.556968 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.556949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34534fe9-e4f9-406f-8f86-6826b20e35bc-cabundle-cert\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.558400 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.558367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-dshm\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.558695 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.558679    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34534fe9-e4f9-406f-8f86-6826b20e35bc-tls-certs\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.566343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.566319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b75rs\" (UniqueName: \"kubernetes.io/projected/34534fe9-e4f9-406f-8f86-6826b20e35bc-kube-api-access-b75rs\") pod \"stop-feature-test-kserve-6c84577c89-kp6lm\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:06.736577 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:06.736489    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:07.013324 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:07.013297    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"]
Sep 16 20:36:07.015659 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:36:07.015626    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34534fe9_e4f9_406f_8f86_6826b20e35bc.slice/crio-b8fb281dd59fa619fbceb2f69f38e6d67d978ecdfdf271544fb3b8230b408355 WatchSource:0}: Error finding container b8fb281dd59fa619fbceb2f69f38e6d67d978ecdfdf271544fb3b8230b408355: Status 404 returned error can't find the container with id b8fb281dd59fa619fbceb2f69f38e6d67d978ecdfdf271544fb3b8230b408355
Sep 16 20:36:07.946176 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:07.946130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" event={"ID":"34534fe9-e4f9-406f-8f86-6826b20e35bc","Type":"ContainerStarted","Data":"fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d"}
Sep 16 20:36:07.946687 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:07.946207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" event={"ID":"34534fe9-e4f9-406f-8f86-6826b20e35bc","Type":"ContainerStarted","Data":"b8fb281dd59fa619fbceb2f69f38e6d67d978ecdfdf271544fb3b8230b408355"}
Sep 16 20:36:09.960154 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:09.960102    2561 generic.go:356] "Generic (PLEG): container finished" podID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerID="fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d" exitCode=0
Sep 16 20:36:09.960559 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:09.960177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" event={"ID":"34534fe9-e4f9-406f-8f86-6826b20e35bc","Type":"ContainerDied","Data":"fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d"}
Sep 16 20:36:10.254745 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:10.254702    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:36:10.254932 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:10.254815    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:36:10.966104 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:10.966070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" event={"ID":"34534fe9-e4f9-406f-8f86-6826b20e35bc","Type":"ContainerStarted","Data":"bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72"}
Sep 16 20:36:10.991385 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:10.991321    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podStartSLOduration=2.8016264250000003 podStartE2EDuration="4.991301807s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.189675382s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:36:10.988437231 +0000 UTC m=+3417.760307642" watchObservedRunningTime="2026-09-16 20:36:10.991301807 +0000 UTC m=+3417.763172218"
Sep 16 20:36:11.736814 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:11.736770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:16.737565 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:16.737521    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:36:16.738884 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:16.738854    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:36:20.254781 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:20.254744    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:36:26.737982 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:26.737928    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:36:28.612755 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:28.612725    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:36:28.739406 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:28.739289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hhjp\" (UniqueName: \"kubernetes.io/projected/61b925a2-4df6-41db-9cc6-5144e02c2aca-kube-api-access-8hhjp\") pod \"61b925a2-4df6-41db-9cc6-5144e02c2aca\" (UID: \"61b925a2-4df6-41db-9cc6-5144e02c2aca\") "
Sep 16 20:36:28.739406 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:28.739349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b925a2-4df6-41db-9cc6-5144e02c2aca-tls-certs\") pod \"61b925a2-4df6-41db-9cc6-5144e02c2aca\" (UID: \"61b925a2-4df6-41db-9cc6-5144e02c2aca\") "
Sep 16 20:36:28.741568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:28.741527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b925a2-4df6-41db-9cc6-5144e02c2aca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61b925a2-4df6-41db-9cc6-5144e02c2aca" (UID: "61b925a2-4df6-41db-9cc6-5144e02c2aca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:36:28.741568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:28.741550    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b925a2-4df6-41db-9cc6-5144e02c2aca-kube-api-access-8hhjp" (OuterVolumeSpecName: "kube-api-access-8hhjp") pod "61b925a2-4df6-41db-9cc6-5144e02c2aca" (UID: "61b925a2-4df6-41db-9cc6-5144e02c2aca"). InnerVolumeSpecName "kube-api-access-8hhjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:36:28.840155 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:28.839961    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b925a2-4df6-41db-9cc6-5144e02c2aca-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:36:28.840155 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:28.840002    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hhjp\" (UniqueName: \"kubernetes.io/projected/61b925a2-4df6-41db-9cc6-5144e02c2aca-kube-api-access-8hhjp\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:36:29.047049 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.047008    2561 generic.go:356] "Generic (PLEG): container finished" podID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerID="86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460" exitCode=0
Sep 16 20:36:29.047240 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.047070    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"
Sep 16 20:36:29.047240 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.047100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" event={"ID":"61b925a2-4df6-41db-9cc6-5144e02c2aca","Type":"ContainerDied","Data":"86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460"}
Sep 16 20:36:29.047240 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.047206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p" event={"ID":"61b925a2-4df6-41db-9cc6-5144e02c2aca","Type":"ContainerDied","Data":"ff862a8bd64b79b50a66ad0e20d2b500443247c4265ab3c1fac711362c5dfcb9"}
Sep 16 20:36:29.047445 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.047242    2561 scope.go:117] "RemoveContainer" containerID="86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460"
Sep 16 20:36:29.059416 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.058058    2561 scope.go:117] "RemoveContainer" containerID="86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460"
Sep 16 20:36:29.059416 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:36:29.059341    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460\": container with ID starting with 86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460 not found: ID does not exist" containerID="86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460"
Sep 16 20:36:29.059526 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.059438    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460"} err="failed to get container status \"86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460\": rpc error: code = NotFound desc = could not find container \"86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460\": container with ID starting with 86ded7ce9e19e18a5a98b68f1134785014e4e014de0215122d4936fb0b7e9460 not found: ID does not exist"
Sep 16 20:36:29.069024 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.068997    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"]
Sep 16 20:36:29.072917 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.072886    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche8v52p"]
Sep 16 20:36:29.683693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:29.683661    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" path="/var/lib/kubelet/pods/61b925a2-4df6-41db-9cc6-5144e02c2aca/volumes"
Sep 16 20:36:36.737370 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:36.737326    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:36:46.737738 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:46.737691    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:36:47.015332 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.015291    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"]
Sep 16 20:36:47.015734 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.015715    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerName="main"
Sep 16 20:36:47.015787 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.015739    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerName="main"
Sep 16 20:36:47.015835 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.015823    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b925a2-4df6-41db-9cc6-5144e02c2aca" containerName="main"
Sep 16 20:36:47.020595 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.020564    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:47.028588 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.028509    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 20:36:47.028848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.028827    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 20:36:47.029308 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.029082    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-8ncwb\""
Sep 16 20:36:47.029308 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.029094    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-g9hhn\""
Sep 16 20:36:47.029308 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.029169    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 20:36:47.031110 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.031086    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"]
Sep 16 20:36:47.087547 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.087508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6v6\" (UniqueName: \"kubernetes.io/projected/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-kube-api-access-cw6v6\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm\" (UID: \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:47.087726 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.087580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm\" (UID: \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:47.187929 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.187896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6v6\" (UniqueName: \"kubernetes.io/projected/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-kube-api-access-cw6v6\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm\" (UID: \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:47.188119 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.187942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm\" (UID: \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:47.190498 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.190467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm\" (UID: \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:47.206903 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.206874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6v6\" (UniqueName: \"kubernetes.io/projected/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-kube-api-access-cw6v6\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm\" (UID: \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:47.333562 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.333473    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:47.615620 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:47.615492    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"]
Sep 16 20:36:47.619369 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:36:47.619337    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebe1b49_c3f7_4e42_833e_89df1019c9f4.slice/crio-b68f2d0f9aa9dcb5dcf41a49b337403a8018156c114f248423c60f216ec67a94 WatchSource:0}: Error finding container b68f2d0f9aa9dcb5dcf41a49b337403a8018156c114f248423c60f216ec67a94: Status 404 returned error can't find the container with id b68f2d0f9aa9dcb5dcf41a49b337403a8018156c114f248423c60f216ec67a94
Sep 16 20:36:48.136600 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:48.136563    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" event={"ID":"1ebe1b49-c3f7-4e42-833e-89df1019c9f4","Type":"ContainerStarted","Data":"23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515"}
Sep 16 20:36:48.137081 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:48.136620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" event={"ID":"1ebe1b49-c3f7-4e42-833e-89df1019c9f4","Type":"ContainerStarted","Data":"b68f2d0f9aa9dcb5dcf41a49b337403a8018156c114f248423c60f216ec67a94"}
Sep 16 20:36:48.137081 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:48.136678    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:36:48.155988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:48.155937    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" podStartSLOduration=2.155923388 podStartE2EDuration="2.155923388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:36:48.155003132 +0000 UTC m=+3454.926873544" watchObservedRunningTime="2026-09-16 20:36:48.155923388 +0000 UTC m=+3454.927793799"
Sep 16 20:36:56.737461 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:36:56.737416    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:37:06.737264 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:37:06.737216    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:37:16.737453 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:37:16.737371    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:37:19.142681 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:37:19.142652    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:37:26.737618 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:37:26.737575    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:37:36.737780 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:37:36.737690    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:37:46.737221 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:37:46.737163    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:37:56.737931 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:37:56.737880    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:38:06.747524 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:06.747494    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:38:06.755417 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:06.755394    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:38:08.223273 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:08.223237    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"]
Sep 16 20:38:23.507882 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:23.507815    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" containerID="cri-o://bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72" gracePeriod=45
Sep 16 20:38:23.748173 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:23.748122    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 20:38:24.251936 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.251909    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:38:24.362331 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362227    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-kserve-provision-location\") pod \"34534fe9-e4f9-406f-8f86-6826b20e35bc\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") "
Sep 16 20:38:24.362331 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-tmp-dir\") pod \"34534fe9-e4f9-406f-8f86-6826b20e35bc\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") "
Sep 16 20:38:24.362331 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362310    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-model-cache\") pod \"34534fe9-e4f9-406f-8f86-6826b20e35bc\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") "
Sep 16 20:38:24.362647 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-home\") pod \"34534fe9-e4f9-406f-8f86-6826b20e35bc\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") "
Sep 16 20:38:24.362647 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362423    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b75rs\" (UniqueName: \"kubernetes.io/projected/34534fe9-e4f9-406f-8f86-6826b20e35bc-kube-api-access-b75rs\") pod \"34534fe9-e4f9-406f-8f86-6826b20e35bc\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") "
Sep 16 20:38:24.362647 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34534fe9-e4f9-406f-8f86-6826b20e35bc-tls-certs\") pod \"34534fe9-e4f9-406f-8f86-6826b20e35bc\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") "
Sep 16 20:38:24.362647 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34534fe9-e4f9-406f-8f86-6826b20e35bc-cabundle-cert\") pod \"34534fe9-e4f9-406f-8f86-6826b20e35bc\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") "
Sep 16 20:38:24.362647 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-dshm\") pod \"34534fe9-e4f9-406f-8f86-6826b20e35bc\" (UID: \"34534fe9-e4f9-406f-8f86-6826b20e35bc\") "
Sep 16 20:38:24.362902 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.362779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "34534fe9-e4f9-406f-8f86-6826b20e35bc" (UID: "34534fe9-e4f9-406f-8f86-6826b20e35bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:38:24.363438 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.363409    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-home" (OuterVolumeSpecName: "home") pod "34534fe9-e4f9-406f-8f86-6826b20e35bc" (UID: "34534fe9-e4f9-406f-8f86-6826b20e35bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:38:24.363596 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.363483    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34534fe9-e4f9-406f-8f86-6826b20e35bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34534fe9-e4f9-406f-8f86-6826b20e35bc" (UID: "34534fe9-e4f9-406f-8f86-6826b20e35bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:38:24.365221 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.365190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34534fe9-e4f9-406f-8f86-6826b20e35bc-kube-api-access-b75rs" (OuterVolumeSpecName: "kube-api-access-b75rs") pod "34534fe9-e4f9-406f-8f86-6826b20e35bc" (UID: "34534fe9-e4f9-406f-8f86-6826b20e35bc"). InnerVolumeSpecName "kube-api-access-b75rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:38:24.365347 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.365325    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-dshm" (OuterVolumeSpecName: "dshm") pod "34534fe9-e4f9-406f-8f86-6826b20e35bc" (UID: "34534fe9-e4f9-406f-8f86-6826b20e35bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:38:24.365443 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.365421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34534fe9-e4f9-406f-8f86-6826b20e35bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34534fe9-e4f9-406f-8f86-6826b20e35bc" (UID: "34534fe9-e4f9-406f-8f86-6826b20e35bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:38:24.374707 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.374677    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34534fe9-e4f9-406f-8f86-6826b20e35bc" (UID: "34534fe9-e4f9-406f-8f86-6826b20e35bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:38:24.390469 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.390444    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34534fe9-e4f9-406f-8f86-6826b20e35bc" (UID: "34534fe9-e4f9-406f-8f86-6826b20e35bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:38:24.463752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.463710    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34534fe9-e4f9-406f-8f86-6826b20e35bc-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:38:24.463752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.463746    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:38:24.463752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.463761    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:38:24.464003 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.463775    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:38:24.464003 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.463789    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:38:24.464003 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.463799    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34534fe9-e4f9-406f-8f86-6826b20e35bc-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:38:24.464003 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.463809    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b75rs\" (UniqueName: \"kubernetes.io/projected/34534fe9-e4f9-406f-8f86-6826b20e35bc-kube-api-access-b75rs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:38:24.464003 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.463820    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34534fe9-e4f9-406f-8f86-6826b20e35bc-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:38:24.554148 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.554115    2561 generic.go:356] "Generic (PLEG): container finished" podID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerID="bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72" exitCode=0
Sep 16 20:38:24.554569 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.554168    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"
Sep 16 20:38:24.554569 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.554183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" event={"ID":"34534fe9-e4f9-406f-8f86-6826b20e35bc","Type":"ContainerDied","Data":"bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72"}
Sep 16 20:38:24.554569 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.554231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm" event={"ID":"34534fe9-e4f9-406f-8f86-6826b20e35bc","Type":"ContainerDied","Data":"b8fb281dd59fa619fbceb2f69f38e6d67d978ecdfdf271544fb3b8230b408355"}
Sep 16 20:38:24.554569 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.554259    2561 scope.go:117] "RemoveContainer" containerID="bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72"
Sep 16 20:38:24.563141 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.563120    2561 scope.go:117] "RemoveContainer" containerID="fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d"
Sep 16 20:38:24.570571 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.570553    2561 scope.go:117] "RemoveContainer" containerID="bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72"
Sep 16 20:38:24.570818 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:38:24.570801    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72\": container with ID starting with bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72 not found: ID does not exist" containerID="bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72"
Sep 16 20:38:24.570872 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.570826    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72"} err="failed to get container status \"bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72\": rpc error: code = NotFound desc = could not find container \"bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72\": container with ID starting with bca551e62c356b174bae795abc051e872ea4e0f408194cfeafb4ccde863c9b72 not found: ID does not exist"
Sep 16 20:38:24.570872 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.570843    2561 scope.go:117] "RemoveContainer" containerID="fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d"
Sep 16 20:38:24.571070 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:38:24.571055    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d\": container with ID starting with fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d not found: ID does not exist" containerID="fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d"
Sep 16 20:38:24.571112 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.571074    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d"} err="failed to get container status \"fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d\": rpc error: code = NotFound desc = could not find container \"fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d\": container with ID starting with fb680693ec6528a7cb94c9c98f0bdcb798ccae29874f5972e68974fce733387d not found: ID does not exist"
Sep 16 20:38:24.576216 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.576194    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"]
Sep 16 20:38:24.579190 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:24.579167    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-kp6lm"]
Sep 16 20:38:25.682522 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:25.682489    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" path="/var/lib/kubelet/pods/34534fe9-e4f9-406f-8f86-6826b20e35bc/volumes"
Sep 16 20:38:52.707993 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:52.707957    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"]
Sep 16 20:38:59.142891 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:38:59.142847    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:39:07.728277 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:07.728166    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerName="main" containerID="cri-o://23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515" gracePeriod=45
Sep 16 20:39:09.142596 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:09.142559    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:39:13.875680 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:13.875647    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:39:13.885061 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:13.885037    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:39:19.142491 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:19.142446    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:39:19.142869 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:19.142556    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:39:29.142530 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:29.142488    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:39:37.970764 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:37.970735    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:39:38.019803 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.019778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-tls-certs\") pod \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\" (UID: \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\") "
Sep 16 20:39:38.019955 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.019845    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw6v6\" (UniqueName: \"kubernetes.io/projected/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-kube-api-access-cw6v6\") pod \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\" (UID: \"1ebe1b49-c3f7-4e42-833e-89df1019c9f4\") "
Sep 16 20:39:38.021995 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.021967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ebe1b49-c3f7-4e42-833e-89df1019c9f4" (UID: "1ebe1b49-c3f7-4e42-833e-89df1019c9f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:39:38.022109 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.022014    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-kube-api-access-cw6v6" (OuterVolumeSpecName: "kube-api-access-cw6v6") pod "1ebe1b49-c3f7-4e42-833e-89df1019c9f4" (UID: "1ebe1b49-c3f7-4e42-833e-89df1019c9f4"). InnerVolumeSpecName "kube-api-access-cw6v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:39:38.120396 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.120342    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:39:38.120396 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.120369    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw6v6\" (UniqueName: \"kubernetes.io/projected/1ebe1b49-c3f7-4e42-833e-89df1019c9f4-kube-api-access-cw6v6\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:39:38.844362 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.844327    2561 generic.go:356] "Generic (PLEG): container finished" podID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerID="23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515" exitCode=0
Sep 16 20:39:38.844362 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.844370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" event={"ID":"1ebe1b49-c3f7-4e42-833e-89df1019c9f4","Type":"ContainerDied","Data":"23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515"}
Sep 16 20:39:38.844635 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.844436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm" event={"ID":"1ebe1b49-c3f7-4e42-833e-89df1019c9f4","Type":"ContainerDied","Data":"b68f2d0f9aa9dcb5dcf41a49b337403a8018156c114f248423c60f216ec67a94"}
Sep 16 20:39:38.844635 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.844389    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"
Sep 16 20:39:38.844635 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.844464    2561 scope.go:117] "RemoveContainer" containerID="23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515"
Sep 16 20:39:38.853369 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.853348    2561 scope.go:117] "RemoveContainer" containerID="23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515"
Sep 16 20:39:38.853654 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:39:38.853631    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515\": container with ID starting with 23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515 not found: ID does not exist" containerID="23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515"
Sep 16 20:39:38.853724 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.853662    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515"} err="failed to get container status \"23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515\": rpc error: code = NotFound desc = could not find container \"23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515\": container with ID starting with 23f2a99d0ec28403532084b87c29535031b27f098278d0e544d0d4bd1ffe0515 not found: ID does not exist"
Sep 16 20:39:38.865716 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.865686    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"]
Sep 16 20:39:38.869362 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:38.869335    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b7767jwnhm"]
Sep 16 20:39:39.683635 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:39.683599    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" path="/var/lib/kubelet/pods/1ebe1b49-c3f7-4e42-833e-89df1019c9f4/volumes"
Sep 16 20:39:55.874444 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874409    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"]
Sep 16 20:39:55.874842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874666    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="storage-initializer"
Sep 16 20:39:55.874842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874677    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="storage-initializer"
Sep 16 20:39:55.874842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874699    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerName="main"
Sep 16 20:39:55.874842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874705    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerName="main"
Sep 16 20:39:55.874842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874714    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main"
Sep 16 20:39:55.874842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874720    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main"
Sep 16 20:39:55.874842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874766    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34534fe9-e4f9-406f-8f86-6826b20e35bc" containerName="main"
Sep 16 20:39:55.874842 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.874779    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebe1b49-c3f7-4e42-833e-89df1019c9f4" containerName="main"
Sep 16 20:39:55.878973 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.878953    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:55.882946 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.882917    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 20:39:55.883077 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.883048    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 20:39:55.883965 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.883947    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 20:39:55.884179 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.884140    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:39:55.884294 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.884274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 20:39:55.884552 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.884528    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-cshgf\""
Sep 16 20:39:55.910261 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.910232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"]
Sep 16 20:39:55.960980 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.960945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-home\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:55.960980 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.960978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tls-certs\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:55.961227 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.961057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-dshm\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:55.961227 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.961089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xsc\" (UniqueName: \"kubernetes.io/projected/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kube-api-access-b8xsc\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:55.961227 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.961117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-model-cache\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:55.961227 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.961141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kserve-provision-location\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:55.961227 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.961173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tmp-dir\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:55.961227 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:55.961189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-cabundle-cert\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.061800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.061762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tmp-dir\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.061800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.061800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-cabundle-cert\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062052 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.061850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-home\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062052 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.061874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tls-certs\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062052 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.061921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-dshm\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062052 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.061947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xsc\" (UniqueName: \"kubernetes.io/projected/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kube-api-access-b8xsc\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062052 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.061978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-model-cache\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062052 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.062001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kserve-provision-location\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062405 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.062201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tmp-dir\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062405 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.062214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-home\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062405 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.062322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-model-cache\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062405 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.062364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kserve-provision-location\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.062810 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.062793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-cabundle-cert\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.064427 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.064405    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-dshm\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.064573 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.064554    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tls-certs\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.071484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.071463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xsc\" (UniqueName: \"kubernetes.io/projected/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kube-api-access-b8xsc\") pod \"router-with-refs-test-kserve-6495b7d567-wnb9t\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.189287 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.189202    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:39:56.471415 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.471314    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"]
Sep 16 20:39:56.474179 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:39:56.474147    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a49e0a4_08d9_43f7_bb4a_14542a2a010a.slice/crio-479c7d1be9d68ce24a39e830579b0e3cd68a3131442a0abe2766949497b1d7d3 WatchSource:0}: Error finding container 479c7d1be9d68ce24a39e830579b0e3cd68a3131442a0abe2766949497b1d7d3: Status 404 returned error can't find the container with id 479c7d1be9d68ce24a39e830579b0e3cd68a3131442a0abe2766949497b1d7d3
Sep 16 20:39:56.921525 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.921476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" event={"ID":"1a49e0a4-08d9-43f7-bb4a-14542a2a010a","Type":"ContainerStarted","Data":"013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793"}
Sep 16 20:39:56.922020 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:56.921541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" event={"ID":"1a49e0a4-08d9-43f7-bb4a-14542a2a010a","Type":"ContainerStarted","Data":"479c7d1be9d68ce24a39e830579b0e3cd68a3131442a0abe2766949497b1d7d3"}
Sep 16 20:39:58.932114 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:58.932080    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerID="013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793" exitCode=0
Sep 16 20:39:58.932475 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:58.932159    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" event={"ID":"1a49e0a4-08d9-43f7-bb4a-14542a2a010a","Type":"ContainerDied","Data":"013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793"}
Sep 16 20:39:59.938115 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:59.938073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" event={"ID":"1a49e0a4-08d9-43f7-bb4a-14542a2a010a","Type":"ContainerStarted","Data":"dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234"}
Sep 16 20:39:59.963155 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:39:59.963098    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podStartSLOduration=2.701236013 podStartE2EDuration="4.963080864s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.261844851s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:39:59.960796265 +0000 UTC m=+3646.732666675" watchObservedRunningTime="2026-09-16 20:39:59.963080864 +0000 UTC m=+3646.734951321"
Sep 16 20:40:00.190198 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:40:00.190106    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:40:06.189482 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:40:06.189438    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:40:06.191106 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:40:06.191076    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:40:16.190487 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:40:16.190440    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:40:26.190178 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:40:26.190138    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:40:36.190006 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:40:36.189912    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:40:46.190555 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:40:46.190508    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:40:56.190037 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:40:56.189988    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:41:06.189956 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:41:06.189902    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:41:16.190478 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:41:16.190429    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:41:26.190397 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:41:26.190336    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:41:36.190560 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:41:36.190510    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:41:46.190330 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:41:46.190283    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 20:41:56.199411 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:41:56.199350    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:41:56.206919 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:41:56.206893    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:42:01.426166 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:01.426108    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"]
Sep 16 20:42:16.448703 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:16.448607    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main" containerID="cri-o://dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234" gracePeriod=45
Sep 16 20:42:17.187494 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.187468    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:42:17.272528 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272497    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-dshm\") pod \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") "
Sep 16 20:42:17.272711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-model-cache\") pod \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") "
Sep 16 20:42:17.272711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272595    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tmp-dir\") pod \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") "
Sep 16 20:42:17.272711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272623    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tls-certs\") pod \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") "
Sep 16 20:42:17.272711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-cabundle-cert\") pod \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") "
Sep 16 20:42:17.272711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272682    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8xsc\" (UniqueName: \"kubernetes.io/projected/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kube-api-access-b8xsc\") pod \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") "
Sep 16 20:42:17.272986 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272825    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a49e0a4-08d9-43f7-bb4a-14542a2a010a" (UID: "1a49e0a4-08d9-43f7-bb4a-14542a2a010a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:42:17.272986 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272883    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-home\") pod \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") "
Sep 16 20:42:17.272986 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.272919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kserve-provision-location\") pod \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\" (UID: \"1a49e0a4-08d9-43f7-bb4a-14542a2a010a\") "
Sep 16 20:42:17.273141 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.273012    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a49e0a4-08d9-43f7-bb4a-14542a2a010a" (UID: "1a49e0a4-08d9-43f7-bb4a-14542a2a010a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:42:17.273353 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.273278    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:42:17.273353 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.273300    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:42:17.273632 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.273601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-home" (OuterVolumeSpecName: "home") pod "1a49e0a4-08d9-43f7-bb4a-14542a2a010a" (UID: "1a49e0a4-08d9-43f7-bb4a-14542a2a010a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:42:17.275110 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.275086    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-dshm" (OuterVolumeSpecName: "dshm") pod "1a49e0a4-08d9-43f7-bb4a-14542a2a010a" (UID: "1a49e0a4-08d9-43f7-bb4a-14542a2a010a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:42:17.275421 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.275390    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a49e0a4-08d9-43f7-bb4a-14542a2a010a" (UID: "1a49e0a4-08d9-43f7-bb4a-14542a2a010a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:42:17.275533 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.275458    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kube-api-access-b8xsc" (OuterVolumeSpecName: "kube-api-access-b8xsc") pod "1a49e0a4-08d9-43f7-bb4a-14542a2a010a" (UID: "1a49e0a4-08d9-43f7-bb4a-14542a2a010a"). InnerVolumeSpecName "kube-api-access-b8xsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:42:17.284902 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.284879    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a49e0a4-08d9-43f7-bb4a-14542a2a010a" (UID: "1a49e0a4-08d9-43f7-bb4a-14542a2a010a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:42:17.295316 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.295295    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a49e0a4-08d9-43f7-bb4a-14542a2a010a" (UID: "1a49e0a4-08d9-43f7-bb4a-14542a2a010a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:42:17.374065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.374029    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:42:17.374065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.374064    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8xsc\" (UniqueName: \"kubernetes.io/projected/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kube-api-access-b8xsc\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:42:17.374252 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.374074    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:42:17.374252 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.374085    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:42:17.374252 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.374095    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:42:17.374252 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.374103    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a49e0a4-08d9-43f7-bb4a-14542a2a010a-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:42:17.522017 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.521979    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerID="dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234" exitCode=0
Sep 16 20:42:17.522603 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.522034    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"
Sep 16 20:42:17.522603 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.522053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" event={"ID":"1a49e0a4-08d9-43f7-bb4a-14542a2a010a","Type":"ContainerDied","Data":"dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234"}
Sep 16 20:42:17.522603 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.522102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t" event={"ID":"1a49e0a4-08d9-43f7-bb4a-14542a2a010a","Type":"ContainerDied","Data":"479c7d1be9d68ce24a39e830579b0e3cd68a3131442a0abe2766949497b1d7d3"}
Sep 16 20:42:17.522603 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.522124    2561 scope.go:117] "RemoveContainer" containerID="dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234"
Sep 16 20:42:17.531132 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.531114    2561 scope.go:117] "RemoveContainer" containerID="013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793"
Sep 16 20:42:17.538697 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.538678    2561 scope.go:117] "RemoveContainer" containerID="dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234"
Sep 16 20:42:17.538979 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:42:17.538959    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234\": container with ID starting with dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234 not found: ID does not exist" containerID="dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234"
Sep 16 20:42:17.539030 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.538987    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234"} err="failed to get container status \"dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234\": rpc error: code = NotFound desc = could not find container \"dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234\": container with ID starting with dca1d2be045ddcb5cf8457cf74997e95be27a8ee80d07788042bebff12837234 not found: ID does not exist"
Sep 16 20:42:17.539030 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.539005    2561 scope.go:117] "RemoveContainer" containerID="013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793"
Sep 16 20:42:17.539234 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:42:17.539218    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793\": container with ID starting with 013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793 not found: ID does not exist" containerID="013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793"
Sep 16 20:42:17.539276 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.539239    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793"} err="failed to get container status \"013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793\": rpc error: code = NotFound desc = could not find container \"013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793\": container with ID starting with 013495d4134543e8f16293807e22f5e868afb63c065e61d0c0c503e306075793 not found: ID does not exist"
Sep 16 20:42:17.544701 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.544676    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"]
Sep 16 20:42:17.550522 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.550503    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6495b7d567-wnb9t"]
Sep 16 20:42:17.682901 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:17.682863    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" path="/var/lib/kubelet/pods/1a49e0a4-08d9-43f7-bb4a-14542a2a010a/volumes"
Sep 16 20:42:56.413793 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.413753    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"]
Sep 16 20:42:56.414192 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.414017    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="storage-initializer"
Sep 16 20:42:56.414192 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.414028    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="storage-initializer"
Sep 16 20:42:56.414192 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.414045    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main"
Sep 16 20:42:56.414192 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.414050    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main"
Sep 16 20:42:56.414192 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.414100    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a49e0a4-08d9-43f7-bb4a-14542a2a010a" containerName="main"
Sep 16 20:42:56.416968 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.416949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.419540 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.419514    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 20:42:56.419808 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.419787    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-bzntk\""
Sep 16 20:42:56.420081 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.420058    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 20:42:56.420211 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.420069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 20:42:56.420211 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.420161    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:42:56.420331 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.420317    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 20:42:56.421401 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.421365    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-lq24g\""
Sep 16 20:42:56.439740 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.439712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"]
Sep 16 20:42:56.456551 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.456521    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"]
Sep 16 20:42:56.459657 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.459612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.475068 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.475037    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"]
Sep 16 20:42:56.560305 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.560305 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb1029cd-b2a2-4706-b874-c944807e9ff4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.560602 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.560602 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.560602 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.560602 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.560602 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.560602 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a274e510-e5be-4239-b947-8b9b88e3fc20-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.560822 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274e510-e5be-4239-b947-8b9b88e3fc20-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.560822 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.560822 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwgnd\" (UniqueName: \"kubernetes.io/projected/eb1029cd-b2a2-4706-b874-c944807e9ff4-kube-api-access-cwgnd\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.560822 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.560822 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpr4\" (UniqueName: \"kubernetes.io/projected/a274e510-e5be-4239-b947-8b9b88e3fc20-kube-api-access-tnpr4\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.560822 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560772    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.560822 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.561044 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.560833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1029cd-b2a2-4706-b874-c944807e9ff4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.661729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.661687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1029cd-b2a2-4706-b874-c944807e9ff4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.661729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.661732    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.661988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.661749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb1029cd-b2a2-4706-b874-c944807e9ff4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.661988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.661787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.661988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.661815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.661988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.661839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.661988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.661862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.661984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a274e510-e5be-4239-b947-8b9b88e3fc20-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274e510-e5be-4239-b947-8b9b88e3fc20-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwgnd\" (UniqueName: \"kubernetes.io/projected/eb1029cd-b2a2-4706-b874-c944807e9ff4-kube-api-access-cwgnd\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpr4\" (UniqueName: \"kubernetes.io/projected/a274e510-e5be-4239-b947-8b9b88e3fc20-kube-api-access-tnpr4\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.662256 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662656    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb1029cd-b2a2-4706-b874-c944807e9ff4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.662930    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.663071 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.663040    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.663518 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.663496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a274e510-e5be-4239-b947-8b9b88e3fc20-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.664594 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.664544    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.664684 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.664663    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.664999 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.664982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1029cd-b2a2-4706-b874-c944807e9ff4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.665061 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.664986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274e510-e5be-4239-b947-8b9b88e3fc20-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.671061 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.671037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwgnd\" (UniqueName: \"kubernetes.io/projected/eb1029cd-b2a2-4706-b874-c944807e9ff4-kube-api-access-cwgnd\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:56.672717 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.672692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpr4\" (UniqueName: \"kubernetes.io/projected/a274e510-e5be-4239-b947-8b9b88e3fc20-kube-api-access-tnpr4\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.727704 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.727656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:56.770454 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:56.770100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:42:57.053113 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:57.053077    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"]
Sep 16 20:42:57.055888 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:42:57.055856    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda274e510_e5be_4239_b947_8b9b88e3fc20.slice/crio-0017118bad1a56c16b9266a1c8adc822d10d18df932cebf6ce205bda3797fe20 WatchSource:0}: Error finding container 0017118bad1a56c16b9266a1c8adc822d10d18df932cebf6ce205bda3797fe20: Status 404 returned error can't find the container with id 0017118bad1a56c16b9266a1c8adc822d10d18df932cebf6ce205bda3797fe20
Sep 16 20:42:57.057883 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:57.057861    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:42:57.098056 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:57.098017    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"]
Sep 16 20:42:57.100994 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:42:57.100967    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1029cd_b2a2_4706_b874_c944807e9ff4.slice/crio-c713d4f5567c3818d9242a766c30f0963ee4f005c10619cbf47a51b55e38245b WatchSource:0}: Error finding container c713d4f5567c3818d9242a766c30f0963ee4f005c10619cbf47a51b55e38245b: Status 404 returned error can't find the container with id c713d4f5567c3818d9242a766c30f0963ee4f005c10619cbf47a51b55e38245b
Sep 16 20:42:57.687074 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:57.687031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" event={"ID":"a274e510-e5be-4239-b947-8b9b88e3fc20","Type":"ContainerStarted","Data":"0017118bad1a56c16b9266a1c8adc822d10d18df932cebf6ce205bda3797fe20"}
Sep 16 20:42:57.687467 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:57.687097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" event={"ID":"eb1029cd-b2a2-4706-b874-c944807e9ff4","Type":"ContainerStarted","Data":"075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06"}
Sep 16 20:42:57.687467 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:57.687133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" event={"ID":"eb1029cd-b2a2-4706-b874-c944807e9ff4","Type":"ContainerStarted","Data":"c713d4f5567c3818d9242a766c30f0963ee4f005c10619cbf47a51b55e38245b"}
Sep 16 20:42:58.691093 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:58.691057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" event={"ID":"a274e510-e5be-4239-b947-8b9b88e3fc20","Type":"ContainerStarted","Data":"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769"}
Sep 16 20:42:58.691500 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:58.691416    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:42:59.698369 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:59.698329    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerID="075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06" exitCode=0
Sep 16 20:42:59.698797 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:59.698414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" event={"ID":"eb1029cd-b2a2-4706-b874-c944807e9ff4","Type":"ContainerDied","Data":"075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06"}
Sep 16 20:42:59.700614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:42:59.700588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" event={"ID":"a274e510-e5be-4239-b947-8b9b88e3fc20","Type":"ContainerStarted","Data":"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635"}
Sep 16 20:43:00.707207 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:00.707165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" event={"ID":"eb1029cd-b2a2-4706-b874-c944807e9ff4","Type":"ContainerStarted","Data":"61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725"}
Sep 16 20:43:00.732910 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:00.732849    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podStartSLOduration=2.31574221 podStartE2EDuration="4.732831844s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.417089634s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:43:00.73093702 +0000 UTC m=+3827.502807435" watchObservedRunningTime="2026-09-16 20:43:00.732831844 +0000 UTC m=+3827.504702255"
Sep 16 20:43:01.715317 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:01.715273    2561 generic.go:356] "Generic (PLEG): container finished" podID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerID="08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635" exitCode=0
Sep 16 20:43:01.715317 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:01.715300    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" event={"ID":"a274e510-e5be-4239-b947-8b9b88e3fc20","Type":"ContainerDied","Data":"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635"}
Sep 16 20:43:02.722095 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:02.722060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" event={"ID":"a274e510-e5be-4239-b947-8b9b88e3fc20","Type":"ContainerStarted","Data":"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d"}
Sep 16 20:43:02.748516 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:02.748447    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podStartSLOduration=2.929957877 podStartE2EDuration="6.748428665s" totalImagesPullingTime="1.462149256s" totalInitContainerRuntime="2.356321532s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:43:02.746113932 +0000 UTC m=+3829.517984368" watchObservedRunningTime="2026-09-16 20:43:02.748428665 +0000 UTC m=+3829.520299077"
Sep 16 20:43:06.728292 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:06.728244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:43:06.728821 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:06.728325    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:43:06.729722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:06.729687    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:43:06.771069 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:06.771028    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:43:06.771265 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:06.771110    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:43:06.772410 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:06.772358    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:43:16.729250 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:16.729196    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:43:16.742418 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:16.742367    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:43:16.771032 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:16.770992    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:43:26.728894 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:26.728848    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:43:26.770981 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:26.770935    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:43:36.729015 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:36.728903    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:43:36.770956 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:36.770902    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:43:46.728394 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:46.728326    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:43:46.770740 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:46.770694    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:43:56.728614 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:56.728553    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:43:56.770876 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:43:56.770829    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:44:06.728889 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:06.728838    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:44:06.771181 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:06.771139    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:44:13.902633 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:13.902603    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:44:13.912008 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:13.911984    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:44:16.728826 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:16.728765    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:44:16.770996 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:16.770943    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:44:26.728717 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:26.728667    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:44:26.770775 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:26.770727    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:44:36.728839 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:36.728784    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:44:36.771457 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:36.771415    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:44:46.728938 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:46.728881    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:44:46.771468 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:46.771421    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:44:56.728455 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:56.728401    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:44:56.770975 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:44:56.770917    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:45:06.728835 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:06.728721    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:45:06.771465 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:06.771416    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:45:16.729038 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:16.728981    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:45:16.771494 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:16.771444    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:45:26.728754 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:26.728699    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:45:26.771208 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:26.771162    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:45:36.728961 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:36.728901    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 16 20:45:36.780849 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:36.780816    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:45:36.788622 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:36.788591    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:45:46.743763 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:46.743729    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:45:47.487279 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:47.487250    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:45:56.973167 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:56.973127    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"]
Sep 16 20:45:56.974789 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:45:56.974759    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"]
Sep 16 20:46:11.995260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:11.995194    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main" containerID="cri-o://61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725" gracePeriod=45
Sep 16 20:46:11.998218 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:11.995874    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main" containerID="cri-o://4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d" gracePeriod=45
Sep 16 20:46:12.813342 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.813294    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="llm-d-routing-sidecar" containerID="cri-o://41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769" gracePeriod=45
Sep 16 20:46:12.947418 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.947390    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:46:12.995782 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.995727    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-kserve-provision-location\") pod \"eb1029cd-b2a2-4706-b874-c944807e9ff4\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") "
Sep 16 20:46:12.995782 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.995760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-tmp-dir\") pod \"eb1029cd-b2a2-4706-b874-c944807e9ff4\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") "
Sep 16 20:46:12.996238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.995802    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1029cd-b2a2-4706-b874-c944807e9ff4-tls-certs\") pod \"eb1029cd-b2a2-4706-b874-c944807e9ff4\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") "
Sep 16 20:46:12.996238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.995833    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-dshm\") pod \"eb1029cd-b2a2-4706-b874-c944807e9ff4\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") "
Sep 16 20:46:12.996238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.995852    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-model-cache\") pod \"eb1029cd-b2a2-4706-b874-c944807e9ff4\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") "
Sep 16 20:46:12.996238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.995880    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwgnd\" (UniqueName: \"kubernetes.io/projected/eb1029cd-b2a2-4706-b874-c944807e9ff4-kube-api-access-cwgnd\") pod \"eb1029cd-b2a2-4706-b874-c944807e9ff4\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") "
Sep 16 20:46:12.996238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.995936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb1029cd-b2a2-4706-b874-c944807e9ff4-cabundle-cert\") pod \"eb1029cd-b2a2-4706-b874-c944807e9ff4\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") "
Sep 16 20:46:12.996238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.995975    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-home\") pod \"eb1029cd-b2a2-4706-b874-c944807e9ff4\" (UID: \"eb1029cd-b2a2-4706-b874-c944807e9ff4\") "
Sep 16 20:46:12.996238 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.996187    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb1029cd-b2a2-4706-b874-c944807e9ff4" (UID: "eb1029cd-b2a2-4706-b874-c944807e9ff4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:12.996672 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.996647    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1029cd-b2a2-4706-b874-c944807e9ff4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb1029cd-b2a2-4706-b874-c944807e9ff4" (UID: "eb1029cd-b2a2-4706-b874-c944807e9ff4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:46:12.996808 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.996785    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-home" (OuterVolumeSpecName: "home") pod "eb1029cd-b2a2-4706-b874-c944807e9ff4" (UID: "eb1029cd-b2a2-4706-b874-c944807e9ff4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:12.999250 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:12.999223    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-dshm" (OuterVolumeSpecName: "dshm") pod "eb1029cd-b2a2-4706-b874-c944807e9ff4" (UID: "eb1029cd-b2a2-4706-b874-c944807e9ff4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:13.000629 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.000598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1029cd-b2a2-4706-b874-c944807e9ff4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb1029cd-b2a2-4706-b874-c944807e9ff4" (UID: "eb1029cd-b2a2-4706-b874-c944807e9ff4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:46:13.000729 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.000695    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1029cd-b2a2-4706-b874-c944807e9ff4-kube-api-access-cwgnd" (OuterVolumeSpecName: "kube-api-access-cwgnd") pod "eb1029cd-b2a2-4706-b874-c944807e9ff4" (UID: "eb1029cd-b2a2-4706-b874-c944807e9ff4"). InnerVolumeSpecName "kube-api-access-cwgnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:46:13.010596 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.010570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb1029cd-b2a2-4706-b874-c944807e9ff4" (UID: "eb1029cd-b2a2-4706-b874-c944807e9ff4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:13.023347 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.023311    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb1029cd-b2a2-4706-b874-c944807e9ff4" (UID: "eb1029cd-b2a2-4706-b874-c944807e9ff4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:13.059752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.059727    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:46:13.096433 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096405    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274e510-e5be-4239-b947-8b9b88e3fc20-tls-certs\") pod \"a274e510-e5be-4239-b947-8b9b88e3fc20\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") "
Sep 16 20:46:13.096610 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a274e510-e5be-4239-b947-8b9b88e3fc20-cabundle-cert\") pod \"a274e510-e5be-4239-b947-8b9b88e3fc20\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") "
Sep 16 20:46:13.096610 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096481    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-dshm\") pod \"a274e510-e5be-4239-b947-8b9b88e3fc20\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") "
Sep 16 20:46:13.096610 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnpr4\" (UniqueName: \"kubernetes.io/projected/a274e510-e5be-4239-b947-8b9b88e3fc20-kube-api-access-tnpr4\") pod \"a274e510-e5be-4239-b947-8b9b88e3fc20\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") "
Sep 16 20:46:13.096791 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096639    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-kserve-provision-location\") pod \"a274e510-e5be-4239-b947-8b9b88e3fc20\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") "
Sep 16 20:46:13.096791 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-model-cache\") pod \"a274e510-e5be-4239-b947-8b9b88e3fc20\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") "
Sep 16 20:46:13.096791 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-home\") pod \"a274e510-e5be-4239-b947-8b9b88e3fc20\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") "
Sep 16 20:46:13.096791 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096741    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-tmp-dir\") pod \"a274e510-e5be-4239-b947-8b9b88e3fc20\" (UID: \"a274e510-e5be-4239-b947-8b9b88e3fc20\") "
Sep 16 20:46:13.097028 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096951    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a274e510-e5be-4239-b947-8b9b88e3fc20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a274e510-e5be-4239-b947-8b9b88e3fc20" (UID: "a274e510-e5be-4239-b947-8b9b88e3fc20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:46:13.097028 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096974    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.097028 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.096994    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.097028 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.097009    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.097028 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.097023    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1029cd-b2a2-4706-b874-c944807e9ff4-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.097294 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.097036    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.097294 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.097048    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1029cd-b2a2-4706-b874-c944807e9ff4-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.097294 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.097062    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwgnd\" (UniqueName: \"kubernetes.io/projected/eb1029cd-b2a2-4706-b874-c944807e9ff4-kube-api-access-cwgnd\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.097294 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.097074    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb1029cd-b2a2-4706-b874-c944807e9ff4-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.097294 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.097163    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-model-cache" (OuterVolumeSpecName: "model-cache") pod "a274e510-e5be-4239-b947-8b9b88e3fc20" (UID: "a274e510-e5be-4239-b947-8b9b88e3fc20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:13.097584 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.097570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-home" (OuterVolumeSpecName: "home") pod "a274e510-e5be-4239-b947-8b9b88e3fc20" (UID: "a274e510-e5be-4239-b947-8b9b88e3fc20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:13.098751 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.098725    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-dshm" (OuterVolumeSpecName: "dshm") pod "a274e510-e5be-4239-b947-8b9b88e3fc20" (UID: "a274e510-e5be-4239-b947-8b9b88e3fc20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:13.099268 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.099240    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a274e510-e5be-4239-b947-8b9b88e3fc20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a274e510-e5be-4239-b947-8b9b88e3fc20" (UID: "a274e510-e5be-4239-b947-8b9b88e3fc20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:46:13.099475 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.099447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a274e510-e5be-4239-b947-8b9b88e3fc20-kube-api-access-tnpr4" (OuterVolumeSpecName: "kube-api-access-tnpr4") pod "a274e510-e5be-4239-b947-8b9b88e3fc20" (UID: "a274e510-e5be-4239-b947-8b9b88e3fc20"). InnerVolumeSpecName "kube-api-access-tnpr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:46:13.117456 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.117424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a274e510-e5be-4239-b947-8b9b88e3fc20" (UID: "a274e510-e5be-4239-b947-8b9b88e3fc20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:13.127140 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.127115    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a274e510-e5be-4239-b947-8b9b88e3fc20" (UID: "a274e510-e5be-4239-b947-8b9b88e3fc20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:13.197417 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.197370    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.197417 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.197414    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274e510-e5be-4239-b947-8b9b88e3fc20-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.197417 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.197423    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a274e510-e5be-4239-b947-8b9b88e3fc20-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.197604 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.197432    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.197604 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.197440    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnpr4\" (UniqueName: \"kubernetes.io/projected/a274e510-e5be-4239-b947-8b9b88e3fc20-kube-api-access-tnpr4\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.197604 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.197448    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.197604 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.197460    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.197604 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.197467    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a274e510-e5be-4239-b947-8b9b88e3fc20-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:13.587317 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.587279    2561 generic.go:356] "Generic (PLEG): container finished" podID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerID="4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d" exitCode=0
Sep 16 20:46:13.587317 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.587310    2561 generic.go:356] "Generic (PLEG): container finished" podID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerID="41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769" exitCode=0
Sep 16 20:46:13.587584 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.587328    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"
Sep 16 20:46:13.587584 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.587362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" event={"ID":"a274e510-e5be-4239-b947-8b9b88e3fc20","Type":"ContainerDied","Data":"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d"}
Sep 16 20:46:13.587584 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.587467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" event={"ID":"a274e510-e5be-4239-b947-8b9b88e3fc20","Type":"ContainerDied","Data":"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769"}
Sep 16 20:46:13.587584 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.587487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2" event={"ID":"a274e510-e5be-4239-b947-8b9b88e3fc20","Type":"ContainerDied","Data":"0017118bad1a56c16b9266a1c8adc822d10d18df932cebf6ce205bda3797fe20"}
Sep 16 20:46:13.587584 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.587521    2561 scope.go:117] "RemoveContainer" containerID="4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d"
Sep 16 20:46:13.589137 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.589115    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerID="61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725" exitCode=0
Sep 16 20:46:13.589137 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.589165    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"
Sep 16 20:46:13.589358 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.589202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" event={"ID":"eb1029cd-b2a2-4706-b874-c944807e9ff4","Type":"ContainerDied","Data":"61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725"}
Sep 16 20:46:13.589358 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.589251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx" event={"ID":"eb1029cd-b2a2-4706-b874-c944807e9ff4","Type":"ContainerDied","Data":"c713d4f5567c3818d9242a766c30f0963ee4f005c10619cbf47a51b55e38245b"}
Sep 16 20:46:13.596717 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.596696    2561 scope.go:117] "RemoveContainer" containerID="08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635"
Sep 16 20:46:13.604901 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.604878    2561 scope.go:117] "RemoveContainer" containerID="41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769"
Sep 16 20:46:13.611621 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.611508    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"]
Sep 16 20:46:13.612664 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.612615    2561 scope.go:117] "RemoveContainer" containerID="4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d"
Sep 16 20:46:13.613096 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:46:13.613068    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d\": container with ID starting with 4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d not found: ID does not exist" containerID="4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d"
Sep 16 20:46:13.613204 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.613106    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d"} err="failed to get container status \"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d\": rpc error: code = NotFound desc = could not find container \"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d\": container with ID starting with 4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d not found: ID does not exist"
Sep 16 20:46:13.613204 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.613141    2561 scope.go:117] "RemoveContainer" containerID="08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635"
Sep 16 20:46:13.613752 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:46:13.613670    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635\": container with ID starting with 08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635 not found: ID does not exist" containerID="08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635"
Sep 16 20:46:13.613752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.613702    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635"} err="failed to get container status \"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635\": rpc error: code = NotFound desc = could not find container \"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635\": container with ID starting with 08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635 not found: ID does not exist"
Sep 16 20:46:13.613752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.613726    2561 scope.go:117] "RemoveContainer" containerID="41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769"
Sep 16 20:46:13.614028 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:46:13.614010    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769\": container with ID starting with 41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769 not found: ID does not exist" containerID="41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769"
Sep 16 20:46:13.614075 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.614035    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769"} err="failed to get container status \"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769\": rpc error: code = NotFound desc = could not find container \"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769\": container with ID starting with 41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769 not found: ID does not exist"
Sep 16 20:46:13.614075 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.614051    2561 scope.go:117] "RemoveContainer" containerID="4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d"
Sep 16 20:46:13.614298 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.614279    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d"} err="failed to get container status \"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d\": rpc error: code = NotFound desc = could not find container \"4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d\": container with ID starting with 4211d4a37c9d2a83015c0307215b15b1675d446d19022f3a6d0649ec5f42d83d not found: ID does not exist"
Sep 16 20:46:13.614342 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.614299    2561 scope.go:117] "RemoveContainer" containerID="08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635"
Sep 16 20:46:13.614568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.614549    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635"} err="failed to get container status \"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635\": rpc error: code = NotFound desc = could not find container \"08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635\": container with ID starting with 08aa556c20bab6fbd981df97cf62000683a78ca1d03a77064194c30ad702c635 not found: ID does not exist"
Sep 16 20:46:13.614568 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.614567    2561 scope.go:117] "RemoveContainer" containerID="41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769"
Sep 16 20:46:13.614851 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.614835    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769"} err="failed to get container status \"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769\": rpc error: code = NotFound desc = could not find container \"41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769\": container with ID starting with 41b13e71e56171b89008a18f93e59f37b31a98d724b525c84cdf78c5adf92769 not found: ID does not exist"
Sep 16 20:46:13.614851 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.614851    2561 scope.go:117] "RemoveContainer" containerID="61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725"
Sep 16 20:46:13.615066 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.615048    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-57b5f6b897d54m2"]
Sep 16 20:46:13.622607 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.622593    2561 scope.go:117] "RemoveContainer" containerID="075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06"
Sep 16 20:46:13.628122 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.628100    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"]
Sep 16 20:46:13.629828 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.629814    2561 scope.go:117] "RemoveContainer" containerID="61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725"
Sep 16 20:46:13.630087 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:46:13.630066    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725\": container with ID starting with 61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725 not found: ID does not exist" containerID="61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725"
Sep 16 20:46:13.630183 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.630089    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725"} err="failed to get container status \"61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725\": rpc error: code = NotFound desc = could not find container \"61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725\": container with ID starting with 61e55dc11b07332aad418b8ac569b313ce7595fecc937b23d4efaefc7c484725 not found: ID does not exist"
Sep 16 20:46:13.630183 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.630104    2561 scope.go:117] "RemoveContainer" containerID="075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06"
Sep 16 20:46:13.630342 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:46:13.630327    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06\": container with ID starting with 075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06 not found: ID does not exist" containerID="075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06"
Sep 16 20:46:13.630402 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.630345    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06"} err="failed to get container status \"075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06\": rpc error: code = NotFound desc = could not find container \"075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06\": container with ID starting with 075b6f91af3c38d1f1094be05ee754135cf1f452225340caf1d3b12c4f63fe06 not found: ID does not exist"
Sep 16 20:46:13.633321 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.633301    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f42pxbx"]
Sep 16 20:46:13.683099 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.683064    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" path="/var/lib/kubelet/pods/a274e510-e5be-4239-b947-8b9b88e3fc20/volumes"
Sep 16 20:46:13.683665 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:13.683641    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" path="/var/lib/kubelet/pods/eb1029cd-b2a2-4706-b874-c944807e9ff4/volumes"
Sep 16 20:46:30.677309 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677275    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"]
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677553    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="storage-initializer"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677565    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="storage-initializer"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677573    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="llm-d-routing-sidecar"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677579    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="llm-d-routing-sidecar"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677596    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="storage-initializer"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677602    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="storage-initializer"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677609    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677614    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677624    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677629    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677669    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="llm-d-routing-sidecar"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677676    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1029cd-b2a2-4706-b874-c944807e9ff4" containerName="main"
Sep 16 20:46:30.677722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.677683    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a274e510-e5be-4239-b947-8b9b88e3fc20" containerName="main"
Sep 16 20:46:30.682118 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.682100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:30.684877 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.684855    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 20:46:30.685928 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.685912    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 16 20:46:30.686000 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.685977    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 20:46:30.686060 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.686043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-2plrq\""
Sep 16 20:46:30.689693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.689667    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"]
Sep 16 20:46:30.725224 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.725194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptc86\" (UniqueName: \"kubernetes.io/projected/14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa-kube-api-access-ptc86\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:30.826518 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.826479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptc86\" (UniqueName: \"kubernetes.io/projected/14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa-kube-api-access-ptc86\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:30.835110 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:30.835083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptc86\" (UniqueName: \"kubernetes.io/projected/14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa-kube-api-access-ptc86\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.143303 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.143264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.149509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.149483    2561 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.243662 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.243628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.243830 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.243772    2561 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.247150 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.247133    2561 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") DevicePath \"csi-0feab911d8edd9fce57f99759cd82b4092b5f313b8684cc2e085a7737e01c176\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.402104 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.402030    2561 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/74953cf68d69e61d780272c3a3e84f8d087162c0b6ab8fd3493c7738a3949f94/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.414274 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.414243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"e2e-pvc-lora-download-d9pqc\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.691516 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.691430    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:33.962779 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:33.962701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"]
Sep 16 20:46:33.965795 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:46:33.965767    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14fb576e_7d3c_4ddc_a7b7_d6eb53e670fa.slice/crio-61e340df0e8db16ed5433d1c5e5da8e509731835557687ba61c28009811766ff WatchSource:0}: Error finding container 61e340df0e8db16ed5433d1c5e5da8e509731835557687ba61c28009811766ff: Status 404 returned error can't find the container with id 61e340df0e8db16ed5433d1c5e5da8e509731835557687ba61c28009811766ff
Sep 16 20:46:34.684132 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:34.684091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc" event={"ID":"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa","Type":"ContainerStarted","Data":"878cab20c26ebf59df33f8aeecbe31bdd4719c724116450ecf4cc93ff9e53476"}
Sep 16 20:46:34.684132 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:34.684138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc" event={"ID":"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa","Type":"ContainerStarted","Data":"61e340df0e8db16ed5433d1c5e5da8e509731835557687ba61c28009811766ff"}
Sep 16 20:46:34.701996 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:34.701939    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc" podStartSLOduration=7.701923597 podStartE2EDuration="7.701923597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:46:34.70069329 +0000 UTC m=+4041.472563701" watchObservedRunningTime="2026-09-16 20:46:34.701923597 +0000 UTC m=+4041.473794009"
Sep 16 20:46:36.694632 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:36.694591    2561 generic.go:356] "Generic (PLEG): container finished" podID="14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa" containerID="878cab20c26ebf59df33f8aeecbe31bdd4719c724116450ecf4cc93ff9e53476" exitCode=0
Sep 16 20:46:36.695013 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:36.694636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc" event={"ID":"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa","Type":"ContainerDied","Data":"878cab20c26ebf59df33f8aeecbe31bdd4719c724116450ecf4cc93ff9e53476"}
Sep 16 20:46:37.830831 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:37.830805    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:37.881996 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:37.881962    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptc86\" (UniqueName: \"kubernetes.io/projected/14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa-kube-api-access-ptc86\") pod \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") "
Sep 16 20:46:37.882170 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:37.882068    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\" (UID: \"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa\") "
Sep 16 20:46:37.884186 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:37.884161    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa-kube-api-access-ptc86" (OuterVolumeSpecName: "kube-api-access-ptc86") pod "14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa" (UID: "14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa"). InnerVolumeSpecName "kube-api-access-ptc86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:46:37.886320 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:37.886302    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9" (OuterVolumeSpecName: "model-storage") pod "14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa" (UID: "14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa"). InnerVolumeSpecName "pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 20:46:37.983194 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:37.983110    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptc86\" (UniqueName: \"kubernetes.io/projected/14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa-kube-api-access-ptc86\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:46:37.983194 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:37.983159    2561 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") on node \"ip-10-0-136-120.ec2.internal\" "
Sep 16 20:46:38.703194 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:38.703160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc" event={"ID":"14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa","Type":"ContainerDied","Data":"61e340df0e8db16ed5433d1c5e5da8e509731835557687ba61c28009811766ff"}
Sep 16 20:46:38.703194 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:38.703187    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"
Sep 16 20:46:38.703194 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:38.703199    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61e340df0e8db16ed5433d1c5e5da8e509731835557687ba61c28009811766ff"
Sep 16 20:46:38.953486 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:38.953420    2561 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9") on node "ip-10-0-136-120.ec2.internal"
Sep 16 20:46:38.991678 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:38.991643    2561 reconciler_common.go:299] "Volume detached for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"csi-0feab911d8edd9fce57f99759cd82b4092b5f313b8684cc2e085a7737e01c176\""
Sep 16 20:46:57.045462 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.045431    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"]
Sep 16 20:46:57.045871 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.045760    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa" containerName="storage-initializer"
Sep 16 20:46:57.045871 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.045775    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa" containerName="storage-initializer"
Sep 16 20:46:57.045871 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.045838    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa" containerName="storage-initializer"
Sep 16 20:46:57.048905 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.048888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.052634 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.052606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:46:57.053014 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.052994    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 20:46:57.053126 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.053040    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-d76zx\""
Sep 16 20:46:57.053530 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.053509    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-dzph5\""
Sep 16 20:46:57.053530 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.053520    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 20:46:57.053698 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.053509    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 20:46:57.053698 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.053519    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 20:46:57.073933 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.073906    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"]
Sep 16 20:46:57.119974 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.119941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-dshm\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.120140 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.119980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.120140 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.120041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.120140 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.120086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142cf16-28d5-4bb2-b26b-409062911a4b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.120140 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.120106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.120286 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.120176    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbcvw\" (UniqueName: \"kubernetes.io/projected/f142cf16-28d5-4bb2-b26b-409062911a4b-kube-api-access-fbcvw\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.120286 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.120207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f142cf16-28d5-4bb2-b26b-409062911a4b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.120286 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.120223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-home\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.220963 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.220926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221134 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.220974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221134 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.220999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142cf16-28d5-4bb2-b26b-409062911a4b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221134 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221134 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbcvw\" (UniqueName: \"kubernetes.io/projected/f142cf16-28d5-4bb2-b26b-409062911a4b-kube-api-access-fbcvw\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221134 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f142cf16-28d5-4bb2-b26b-409062911a4b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221134 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-home\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221134 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-dshm\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221639 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221639 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221754 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221811 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221766    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-home\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.221890 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.221871    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f142cf16-28d5-4bb2-b26b-409062911a4b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.223950 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.223922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-dshm\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.224129 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.224112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142cf16-28d5-4bb2-b26b-409062911a4b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.230366 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.230346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbcvw\" (UniqueName: \"kubernetes.io/projected/f142cf16-28d5-4bb2-b26b-409062911a4b-kube-api-access-fbcvw\") pod \"custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.358440 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.358343    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:57.637143 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.637116    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"]
Sep 16 20:46:57.639698 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:46:57.639666    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf142cf16_28d5_4bb2_b26b_409062911a4b.slice/crio-dc5248b4842f3f9fac30f813fa14e3f3070622a620bfb4619aa86959654ac21b WatchSource:0}: Error finding container dc5248b4842f3f9fac30f813fa14e3f3070622a620bfb4619aa86959654ac21b: Status 404 returned error can't find the container with id dc5248b4842f3f9fac30f813fa14e3f3070622a620bfb4619aa86959654ac21b
Sep 16 20:46:57.780947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.780906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" event={"ID":"f142cf16-28d5-4bb2-b26b-409062911a4b","Type":"ContainerStarted","Data":"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2"}
Sep 16 20:46:57.781141 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.780963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" event={"ID":"f142cf16-28d5-4bb2-b26b-409062911a4b","Type":"ContainerStarted","Data":"dc5248b4842f3f9fac30f813fa14e3f3070622a620bfb4619aa86959654ac21b"}
Sep 16 20:46:57.781141 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:57.781031    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:46:58.787725 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:46:58.787681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" event={"ID":"f142cf16-28d5-4bb2-b26b-409062911a4b","Type":"ContainerStarted","Data":"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34"}
Sep 16 20:47:00.797253 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:00.797217    2561 generic.go:356] "Generic (PLEG): container finished" podID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerID="b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34" exitCode=0
Sep 16 20:47:00.797654 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:00.797258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" event={"ID":"f142cf16-28d5-4bb2-b26b-409062911a4b","Type":"ContainerDied","Data":"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34"}
Sep 16 20:47:01.804661 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:01.804616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" event={"ID":"f142cf16-28d5-4bb2-b26b-409062911a4b","Type":"ContainerStarted","Data":"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282"}
Sep 16 20:47:01.829926 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:01.829850    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podStartSLOduration=2.660992776 podStartE2EDuration="4.829828528s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.168835752s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:47:01.829190113 +0000 UTC m=+4068.601060525" watchObservedRunningTime="2026-09-16 20:47:01.829828528 +0000 UTC m=+4068.601698942"
Sep 16 20:47:07.359458 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:07.359411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:47:07.359975 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:07.359499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:47:07.360673 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:07.360624    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:47:07.372222 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:07.372196    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:47:17.359780 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:17.359726    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:47:27.359771 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:27.359717    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:47:37.359678 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:37.359624    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:47:47.359762 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:47.359708    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:47:57.359092 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:47:57.359037    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:48:07.359777 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:48:07.359664    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:48:17.359045 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:48:17.358995    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:48:27.359686 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:48:27.359635    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:48:37.359263 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:48:37.359211    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:48:47.359924 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:48:47.359866    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8001/health\": dial tcp 10.132.0.66:8001: connect: connection refused"
Sep 16 20:48:57.369391 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:48:57.369343    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:48:57.385492 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:48:57.385466    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:49:06.454034 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:06.453995    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"]
Sep 16 20:49:13.933175 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:13.933145    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:49:13.938662 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:13.938639    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:49:21.477653 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:21.477587    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main" containerID="cri-o://a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282" gracePeriod=45
Sep 16 20:49:22.095230 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.095200    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="llm-d-routing-sidecar" containerID="cri-o://3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2" gracePeriod=45
Sep 16 20:49:22.331837 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.331811    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:49:22.370680 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370646    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-model-cache\") pod \"f142cf16-28d5-4bb2-b26b-409062911a4b\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") "
Sep 16 20:49:22.370848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370685    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-dshm\") pod \"f142cf16-28d5-4bb2-b26b-409062911a4b\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") "
Sep 16 20:49:22.370848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370714    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142cf16-28d5-4bb2-b26b-409062911a4b-tls-certs\") pod \"f142cf16-28d5-4bb2-b26b-409062911a4b\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") "
Sep 16 20:49:22.370848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370737    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-kserve-provision-location\") pod \"f142cf16-28d5-4bb2-b26b-409062911a4b\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") "
Sep 16 20:49:22.370848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370759    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f142cf16-28d5-4bb2-b26b-409062911a4b-cabundle-cert\") pod \"f142cf16-28d5-4bb2-b26b-409062911a4b\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") "
Sep 16 20:49:22.370848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370785    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbcvw\" (UniqueName: \"kubernetes.io/projected/f142cf16-28d5-4bb2-b26b-409062911a4b-kube-api-access-fbcvw\") pod \"f142cf16-28d5-4bb2-b26b-409062911a4b\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") "
Sep 16 20:49:22.370848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-home\") pod \"f142cf16-28d5-4bb2-b26b-409062911a4b\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") "
Sep 16 20:49:22.370848 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370841    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-tmp-dir\") pod \"f142cf16-28d5-4bb2-b26b-409062911a4b\" (UID: \"f142cf16-28d5-4bb2-b26b-409062911a4b\") "
Sep 16 20:49:22.371215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.370965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f142cf16-28d5-4bb2-b26b-409062911a4b" (UID: "f142cf16-28d5-4bb2-b26b-409062911a4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.371215 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.371144    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.371319 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.371295    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f142cf16-28d5-4bb2-b26b-409062911a4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f142cf16-28d5-4bb2-b26b-409062911a4b" (UID: "f142cf16-28d5-4bb2-b26b-409062911a4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:49:22.371691 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.371661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-home" (OuterVolumeSpecName: "home") pod "f142cf16-28d5-4bb2-b26b-409062911a4b" (UID: "f142cf16-28d5-4bb2-b26b-409062911a4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.373497 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.373467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f142cf16-28d5-4bb2-b26b-409062911a4b-kube-api-access-fbcvw" (OuterVolumeSpecName: "kube-api-access-fbcvw") pod "f142cf16-28d5-4bb2-b26b-409062911a4b" (UID: "f142cf16-28d5-4bb2-b26b-409062911a4b"). InnerVolumeSpecName "kube-api-access-fbcvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:49:22.373590 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.373536    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f142cf16-28d5-4bb2-b26b-409062911a4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f142cf16-28d5-4bb2-b26b-409062911a4b" (UID: "f142cf16-28d5-4bb2-b26b-409062911a4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:49:22.373967 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.373948    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-dshm" (OuterVolumeSpecName: "dshm") pod "f142cf16-28d5-4bb2-b26b-409062911a4b" (UID: "f142cf16-28d5-4bb2-b26b-409062911a4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.382938 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.382915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f142cf16-28d5-4bb2-b26b-409062911a4b" (UID: "f142cf16-28d5-4bb2-b26b-409062911a4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.398036 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.398003    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f142cf16-28d5-4bb2-b26b-409062911a4b" (UID: "f142cf16-28d5-4bb2-b26b-409062911a4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.409555 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.409524    2561 generic.go:356] "Generic (PLEG): container finished" podID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerID="a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282" exitCode=0
Sep 16 20:49:22.409555 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.409549    2561 generic.go:356] "Generic (PLEG): container finished" podID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerID="3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2" exitCode=0
Sep 16 20:49:22.409737 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.409591    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"
Sep 16 20:49:22.409737 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.409597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" event={"ID":"f142cf16-28d5-4bb2-b26b-409062911a4b","Type":"ContainerDied","Data":"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282"}
Sep 16 20:49:22.409737 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.409655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" event={"ID":"f142cf16-28d5-4bb2-b26b-409062911a4b","Type":"ContainerDied","Data":"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2"}
Sep 16 20:49:22.409737 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.409673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb" event={"ID":"f142cf16-28d5-4bb2-b26b-409062911a4b","Type":"ContainerDied","Data":"dc5248b4842f3f9fac30f813fa14e3f3070622a620bfb4619aa86959654ac21b"}
Sep 16 20:49:22.409737 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.409694    2561 scope.go:117] "RemoveContainer" containerID="a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282"
Sep 16 20:49:22.418314 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.418289    2561 scope.go:117] "RemoveContainer" containerID="b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34"
Sep 16 20:49:22.425453 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.425434    2561 scope.go:117] "RemoveContainer" containerID="3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2"
Sep 16 20:49:22.432739 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.432720    2561 scope.go:117] "RemoveContainer" containerID="a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282"
Sep 16 20:49:22.432905 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.432886    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"]
Sep 16 20:49:22.433032 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:49:22.433017    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282\": container with ID starting with a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282 not found: ID does not exist" containerID="a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282"
Sep 16 20:49:22.433081 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.433043    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282"} err="failed to get container status \"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282\": rpc error: code = NotFound desc = could not find container \"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282\": container with ID starting with a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282 not found: ID does not exist"
Sep 16 20:49:22.433081 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.433059    2561 scope.go:117] "RemoveContainer" containerID="b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34"
Sep 16 20:49:22.433314 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:49:22.433293    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34\": container with ID starting with b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34 not found: ID does not exist" containerID="b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34"
Sep 16 20:49:22.433359 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.433321    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34"} err="failed to get container status \"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34\": rpc error: code = NotFound desc = could not find container \"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34\": container with ID starting with b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34 not found: ID does not exist"
Sep 16 20:49:22.433359 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.433338    2561 scope.go:117] "RemoveContainer" containerID="3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2"
Sep 16 20:49:22.433628 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:49:22.433609    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2\": container with ID starting with 3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2 not found: ID does not exist" containerID="3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2"
Sep 16 20:49:22.433712 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.433630    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2"} err="failed to get container status \"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2\": rpc error: code = NotFound desc = could not find container \"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2\": container with ID starting with 3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2 not found: ID does not exist"
Sep 16 20:49:22.433712 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.433643    2561 scope.go:117] "RemoveContainer" containerID="a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282"
Sep 16 20:49:22.433947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.433896    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282"} err="failed to get container status \"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282\": rpc error: code = NotFound desc = could not find container \"a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282\": container with ID starting with a9635caadf822e80ad81dcc79e404c00f18ed65f5b519f6d6f467f6a22663282 not found: ID does not exist"
Sep 16 20:49:22.433947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.433923    2561 scope.go:117] "RemoveContainer" containerID="b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34"
Sep 16 20:49:22.434743 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.434614    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34"} err="failed to get container status \"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34\": rpc error: code = NotFound desc = could not find container \"b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34\": container with ID starting with b35db0d5d44b7e225e61823a948ac8f8117450f12b7124596b18d3d748423c34 not found: ID does not exist"
Sep 16 20:49:22.434743 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.434642    2561 scope.go:117] "RemoveContainer" containerID="3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2"
Sep 16 20:49:22.434958 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.434935    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2"} err="failed to get container status \"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2\": rpc error: code = NotFound desc = could not find container \"3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2\": container with ID starting with 3dd17b47be3965c27e53cc418926d5e513766c2a13709ebcaf78bdef7ecff6d2 not found: ID does not exist"
Sep 16 20:49:22.436210 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.436191    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66698c7f78-lwhwb"]
Sep 16 20:49:22.472493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.472468    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbcvw\" (UniqueName: \"kubernetes.io/projected/f142cf16-28d5-4bb2-b26b-409062911a4b-kube-api-access-fbcvw\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.472493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.472492    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.472650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.472503    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.472650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.472512    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.472650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.472520    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142cf16-28d5-4bb2-b26b-409062911a4b-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.472650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.472529    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f142cf16-28d5-4bb2-b26b-409062911a4b-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.472650 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:22.472536    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f142cf16-28d5-4bb2-b26b-409062911a4b-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:49:23.683003 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:49:23.682969    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" path="/var/lib/kubelet/pods/f142cf16-28d5-4bb2-b26b-409062911a4b/volumes"
Sep 16 20:50:06.481638 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:06.481600    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"]
Sep 16 20:50:06.487418 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:06.487393    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-d9pqc"]
Sep 16 20:50:07.682451 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:07.682405    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa" path="/var/lib/kubelet/pods/14fb576e-7d3c-4ddc-a7b7-d6eb53e670fa/volumes"
Sep 16 20:50:17.047083 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047047    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"]
Sep 16 20:50:17.047484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047314    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main"
Sep 16 20:50:17.047484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047326    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main"
Sep 16 20:50:17.047484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047347    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="llm-d-routing-sidecar"
Sep 16 20:50:17.047484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047352    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="llm-d-routing-sidecar"
Sep 16 20:50:17.047484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047358    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="storage-initializer"
Sep 16 20:50:17.047484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047364    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="storage-initializer"
Sep 16 20:50:17.047484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047441    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="llm-d-routing-sidecar"
Sep 16 20:50:17.047484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.047450    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f142cf16-28d5-4bb2-b26b-409062911a4b" containerName="main"
Sep 16 20:50:17.050600 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.050583    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.056929 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.056905    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 20:50:17.057278 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.057257    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 20:50:17.057278 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.057267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 20:50:17.057462 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.057260    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 20:50:17.057462 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.057267    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-26n4k\""
Sep 16 20:50:17.057878 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.057863    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:50:17.061233 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.061217    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-tfs4k\""
Sep 16 20:50:17.078444 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.078414    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"]
Sep 16 20:50:17.092024 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.092002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.092136 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.092033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-model-cache\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.092136 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.092056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-dshm\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.092136 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.092081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kmn\" (UniqueName: \"kubernetes.io/projected/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kube-api-access-v6kmn\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.092136 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.092119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tls-certs\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.092136 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.092133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.092306 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.092160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-home\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.092306 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.092180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.192798 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.192762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-dshm\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.192798 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.192800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kmn\" (UniqueName: \"kubernetes.io/projected/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kube-api-access-v6kmn\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193023 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.192927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tls-certs\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193023 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.192962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193023 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.193002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-home\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193169 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.193040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193169 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.193098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193440 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.193415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.193482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-model-cache\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.193498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-home\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.193587 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.193471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.195369 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.194114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-model-cache\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.195369 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.194320    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.195922 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.195899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tls-certs\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.200020 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.199999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-dshm\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.202232 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.202212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kmn\" (UniqueName: \"kubernetes.io/projected/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kube-api-access-v6kmn\") pod \"router-with-refs-pd-test-kserve-85778cd84-bfklv\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.358967 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.358876    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:17.664084 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:17.664057    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"]
Sep 16 20:50:17.666562 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:50:17.666534    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cb49bf_1ec0_4666_b2fd_bf4acd4df10a.slice/crio-216a545b009e9619d9a68476a5fa96c70caddc54b12791a02079d71eea7efdd8 WatchSource:0}: Error finding container 216a545b009e9619d9a68476a5fa96c70caddc54b12791a02079d71eea7efdd8: Status 404 returned error can't find the container with id 216a545b009e9619d9a68476a5fa96c70caddc54b12791a02079d71eea7efdd8
Sep 16 20:50:18.636516 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:18.636476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" event={"ID":"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a","Type":"ContainerStarted","Data":"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809"}
Sep 16 20:50:18.636907 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:18.636536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" event={"ID":"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a","Type":"ContainerStarted","Data":"216a545b009e9619d9a68476a5fa96c70caddc54b12791a02079d71eea7efdd8"}
Sep 16 20:50:18.636907 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:18.636617    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:19.648999 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:19.648964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" event={"ID":"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a","Type":"ContainerStarted","Data":"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b"}
Sep 16 20:50:21.659246 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:21.659208    2561 generic.go:356] "Generic (PLEG): container finished" podID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerID="52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b" exitCode=0
Sep 16 20:50:21.659637 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:21.659284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" event={"ID":"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a","Type":"ContainerDied","Data":"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b"}
Sep 16 20:50:22.233807 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.233771    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"]
Sep 16 20:50:22.237095 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.237072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.265748 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.265712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"]
Sep 16 20:50:22.432258 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.432220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6b38b4c-25bc-4fad-95b2-12511f20562d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.432493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.432318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-home\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.432493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.432368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-dshm\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.432493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.432424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-model-cache\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.432493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.432448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b38b4c-25bc-4fad-95b2-12511f20562d-tls-certs\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.432493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.432480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jn6v\" (UniqueName: \"kubernetes.io/projected/f6b38b4c-25bc-4fad-95b2-12511f20562d-kube-api-access-5jn6v\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.432762 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.432564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.432762 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.432611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.534150 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.534111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6b38b4c-25bc-4fad-95b2-12511f20562d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.534429 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.534407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-home\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.534579 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.534564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-dshm\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.534693 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.534679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-model-cache\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.534805 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.534791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b38b4c-25bc-4fad-95b2-12511f20562d-tls-certs\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.534917 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.534890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jn6v\" (UniqueName: \"kubernetes.io/projected/f6b38b4c-25bc-4fad-95b2-12511f20562d-kube-api-access-5jn6v\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.535050 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.535035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.535183 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.535170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.535739 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.535715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.536537 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.536515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6b38b4c-25bc-4fad-95b2-12511f20562d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.536938 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.536919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-home\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.539929 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.539741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-dshm\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.540412 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.540156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.540412 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.540287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-model-cache\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.542716 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.542694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b38b4c-25bc-4fad-95b2-12511f20562d-tls-certs\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.548974 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.548949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jn6v\" (UniqueName: \"kubernetes.io/projected/f6b38b4c-25bc-4fad-95b2-12511f20562d-kube-api-access-5jn6v\") pod \"router-with-refs-pd-test-kserve-84b6b568dc-nlhr6\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:22.666596 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.666550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" event={"ID":"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a","Type":"ContainerStarted","Data":"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8"}
Sep 16 20:50:22.692253 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.692197    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podStartSLOduration=3.501221268 podStartE2EDuration="5.692181634s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.190960366s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:50:22.69029267 +0000 UTC m=+4269.462163093" watchObservedRunningTime="2026-09-16 20:50:22.692181634 +0000 UTC m=+4269.464052044"
Sep 16 20:50:22.845620 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:22.845526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:23.128101 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:23.128070    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"]
Sep 16 20:50:23.130870 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:50:23.130837    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b38b4c_25bc_4fad_95b2_12511f20562d.slice/crio-31e8bc3d3c96f4df038dd2b5eea5a7d888c6db01866aa8ae5f7c99b8b5eca655 WatchSource:0}: Error finding container 31e8bc3d3c96f4df038dd2b5eea5a7d888c6db01866aa8ae5f7c99b8b5eca655: Status 404 returned error can't find the container with id 31e8bc3d3c96f4df038dd2b5eea5a7d888c6db01866aa8ae5f7c99b8b5eca655
Sep 16 20:50:23.672107 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:23.672066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" event={"ID":"f6b38b4c-25bc-4fad-95b2-12511f20562d","Type":"ContainerStarted","Data":"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22"}
Sep 16 20:50:23.672527 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:23.672128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" event={"ID":"f6b38b4c-25bc-4fad-95b2-12511f20562d","Type":"ContainerStarted","Data":"31e8bc3d3c96f4df038dd2b5eea5a7d888c6db01866aa8ae5f7c99b8b5eca655"}
Sep 16 20:50:23.672527 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:23.672172    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:24.678107 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:24.678075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" event={"ID":"f6b38b4c-25bc-4fad-95b2-12511f20562d","Type":"ContainerStarted","Data":"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95"}
Sep 16 20:50:26.688819 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:26.688781    2561 generic.go:356] "Generic (PLEG): container finished" podID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerID="47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95" exitCode=0
Sep 16 20:50:26.689253 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:26.688853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" event={"ID":"f6b38b4c-25bc-4fad-95b2-12511f20562d","Type":"ContainerDied","Data":"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95"}
Sep 16 20:50:27.359111 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:27.359066    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:27.359368 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:27.359183    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:27.361068 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:27.361033    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:50:27.396069 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:27.396031    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:50:27.695579 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:27.695484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" event={"ID":"f6b38b4c-25bc-4fad-95b2-12511f20562d","Type":"ContainerStarted","Data":"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf"}
Sep 16 20:50:27.719142 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:27.719083    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podStartSLOduration=3.373261607 podStartE2EDuration="5.719060872s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.345799265s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:50:27.717613518 +0000 UTC m=+4274.489483930" watchObservedRunningTime="2026-09-16 20:50:27.719060872 +0000 UTC m=+4274.490931286"
Sep 16 20:50:32.846414 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:32.846355    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:32.846938 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:32.846487    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:32.847823 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:32.847784    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:50:37.359808 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:37.359749    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:50:42.846395 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:42.846321    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:50:42.859509 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:42.859474    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:50:47.367684 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:47.367628    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:50:52.846830 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:52.846779    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:50:57.360019 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:50:57.359962    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:51:02.846340 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:02.846288    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:51:07.359984 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:07.359876    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:51:08.471199 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.471153    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"]
Sep 16 20:51:08.474393 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.474355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:08.477009 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.476986    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-9vvsb\""
Sep 16 20:51:08.485048 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.485019    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"]
Sep 16 20:51:08.532764 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.532723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c636e4-b347-4e62-961b-06dcf397a69d-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs\" (UID: \"a6c636e4-b347-4e62-961b-06dcf397a69d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:08.532764 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.532769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/a6c636e4-b347-4e62-961b-06dcf397a69d-kube-api-access-phwkn\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs\" (UID: \"a6c636e4-b347-4e62-961b-06dcf397a69d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:08.634115 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.634070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/a6c636e4-b347-4e62-961b-06dcf397a69d-kube-api-access-phwkn\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs\" (UID: \"a6c636e4-b347-4e62-961b-06dcf397a69d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:08.634318 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.634178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c636e4-b347-4e62-961b-06dcf397a69d-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs\" (UID: \"a6c636e4-b347-4e62-961b-06dcf397a69d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:08.636964 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.636932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c636e4-b347-4e62-961b-06dcf397a69d-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs\" (UID: \"a6c636e4-b347-4e62-961b-06dcf397a69d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:08.644397 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.644335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/a6c636e4-b347-4e62-961b-06dcf397a69d-kube-api-access-phwkn\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs\" (UID: \"a6c636e4-b347-4e62-961b-06dcf397a69d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:08.786794 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:08.786754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:09.072438 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:09.072283    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"]
Sep 16 20:51:09.074661 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:51:09.074624    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c636e4_b347_4e62_961b_06dcf397a69d.slice/crio-46500b6c41110268910f1314ba817d1127e596765b2d79d64e29ae96c3ce80fd WatchSource:0}: Error finding container 46500b6c41110268910f1314ba817d1127e596765b2d79d64e29ae96c3ce80fd: Status 404 returned error can't find the container with id 46500b6c41110268910f1314ba817d1127e596765b2d79d64e29ae96c3ce80fd
Sep 16 20:51:09.900562 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:09.900517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" event={"ID":"a6c636e4-b347-4e62-961b-06dcf397a69d","Type":"ContainerStarted","Data":"d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee"}
Sep 16 20:51:09.900958 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:09.900590    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" event={"ID":"a6c636e4-b347-4e62-961b-06dcf397a69d","Type":"ContainerStarted","Data":"46500b6c41110268910f1314ba817d1127e596765b2d79d64e29ae96c3ce80fd"}
Sep 16 20:51:09.900958 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:09.900637    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:09.919018 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:09.918958    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" podStartSLOduration=1.918912607 podStartE2EDuration="1.918912607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:51:09.918363774 +0000 UTC m=+4316.690234191" watchObservedRunningTime="2026-09-16 20:51:09.918912607 +0000 UTC m=+4316.690783019"
Sep 16 20:51:12.846066 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:12.846013    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:51:17.359846 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:17.359786    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:51:22.846731 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:22.846678    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:51:27.360008 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:27.359961    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:51:32.846654 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:32.846601    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:51:37.359613 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:37.359553    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:51:40.907305 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:40.907269    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:51:42.846422 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:42.846358    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:51:47.359545 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:47.359486    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:51:52.846879 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:52.846827    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:51:57.359836 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:51:57.359776    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:52:02.846269 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:02.846226    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:52:07.360148 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:07.360087    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:52:12.846061 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:12.846009    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:52:17.360191 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:17.360141    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:52:22.846173 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:22.846107    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:52:27.360087 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:27.360034    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:52:32.846435 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:32.846356    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:52:37.359944 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:37.359839    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:52:42.846482 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:42.846434    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:52:47.360192 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:47.360027    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 16 20:52:52.845969 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:52.845912    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8001/health\": dial tcp 10.132.0.68:8001: connect: connection refused"
Sep 16 20:52:57.369116 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:57.369084    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:52:57.385641 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:52:57.385609    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:53:02.856567 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:02.856533    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:53:02.868267 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:02.868238    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:53:02.935154 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:02.935115    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"]
Sep 16 20:53:17.958659 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:17.958600    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main" containerID="cri-o://9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8" gracePeriod=45
Sep 16 20:53:18.575133 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.575070    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="llm-d-routing-sidecar" containerID="cri-o://80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809" gracePeriod=45
Sep 16 20:53:18.816724 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.816694    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:53:18.908285 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.908250    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kserve-provision-location\") pod \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") "
Sep 16 20:53:18.908285 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.908284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tmp-dir\") pod \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") "
Sep 16 20:53:18.908529 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.908310    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-model-cache\") pod \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") "
Sep 16 20:53:18.908529 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.908362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-cabundle-cert\") pod \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") "
Sep 16 20:53:18.908529 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.908418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-home\") pod \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") "
Sep 16 20:53:18.908529 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.908451    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6kmn\" (UniqueName: \"kubernetes.io/projected/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kube-api-access-v6kmn\") pod \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") "
Sep 16 20:53:18.908529 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.908481    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-dshm\") pod \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") "
Sep 16 20:53:18.908732 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.908536    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tls-certs\") pod \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\" (UID: \"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a\") "
Sep 16 20:53:18.909038 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.909004    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" (UID: "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:53:18.909544 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.909129    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:18.911544 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.910232    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-home" (OuterVolumeSpecName: "home") pod "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" (UID: "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:18.911544 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.910270    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-model-cache" (OuterVolumeSpecName: "model-cache") pod "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" (UID: "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:18.913413 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.912123    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" (UID: "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:53:18.918416 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.918388    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-dshm" (OuterVolumeSpecName: "dshm") pod "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" (UID: "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:18.919291 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.919241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kube-api-access-v6kmn" (OuterVolumeSpecName: "kube-api-access-v6kmn") pod "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" (UID: "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a"). InnerVolumeSpecName "kube-api-access-v6kmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:53:18.925570 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.925541    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" (UID: "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:18.935037 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:18.935005    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" (UID: "76cb49bf-1ec0-4666-b2fd-bf4acd4df10a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:19.010042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.010004    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:19.010042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.010036    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:19.010473 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.010066    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:19.010473 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.010078    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:19.010473 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.010086    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:19.010473 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.010094    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6kmn\" (UniqueName: \"kubernetes.io/projected/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-kube-api-access-v6kmn\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:19.010473 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.010101    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:19.504173 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.504131    2561 generic.go:356] "Generic (PLEG): container finished" podID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerID="9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8" exitCode=0
Sep 16 20:53:19.504173 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.504166    2561 generic.go:356] "Generic (PLEG): container finished" podID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerID="80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809" exitCode=0
Sep 16 20:53:19.504424 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.504214    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"
Sep 16 20:53:19.504424 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.504224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" event={"ID":"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a","Type":"ContainerDied","Data":"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8"}
Sep 16 20:53:19.504424 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.504334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" event={"ID":"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a","Type":"ContainerDied","Data":"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809"}
Sep 16 20:53:19.504424 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.504366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv" event={"ID":"76cb49bf-1ec0-4666-b2fd-bf4acd4df10a","Type":"ContainerDied","Data":"216a545b009e9619d9a68476a5fa96c70caddc54b12791a02079d71eea7efdd8"}
Sep 16 20:53:19.504424 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.504410    2561 scope.go:117] "RemoveContainer" containerID="9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8"
Sep 16 20:53:19.514493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.514470    2561 scope.go:117] "RemoveContainer" containerID="52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b"
Sep 16 20:53:19.522639 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.522613    2561 scope.go:117] "RemoveContainer" containerID="80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809"
Sep 16 20:53:19.529030 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.529001    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"]
Sep 16 20:53:19.533213 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.533193    2561 scope.go:117] "RemoveContainer" containerID="9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8"
Sep 16 20:53:19.533539 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:53:19.533516    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8\": container with ID starting with 9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8 not found: ID does not exist" containerID="9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8"
Sep 16 20:53:19.533631 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.533547    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8"} err="failed to get container status \"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8\": rpc error: code = NotFound desc = could not find container \"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8\": container with ID starting with 9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8 not found: ID does not exist"
Sep 16 20:53:19.533631 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.533565    2561 scope.go:117] "RemoveContainer" containerID="52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b"
Sep 16 20:53:19.533752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.533706    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-85778cd84-bfklv"]
Sep 16 20:53:19.533814 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:53:19.533798    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b\": container with ID starting with 52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b not found: ID does not exist" containerID="52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b"
Sep 16 20:53:19.533874 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.533818    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b"} err="failed to get container status \"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b\": rpc error: code = NotFound desc = could not find container \"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b\": container with ID starting with 52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b not found: ID does not exist"
Sep 16 20:53:19.533874 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.533831    2561 scope.go:117] "RemoveContainer" containerID="80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809"
Sep 16 20:53:19.534066 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:53:19.534050    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809\": container with ID starting with 80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809 not found: ID does not exist" containerID="80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809"
Sep 16 20:53:19.534125 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.534073    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809"} err="failed to get container status \"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809\": rpc error: code = NotFound desc = could not find container \"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809\": container with ID starting with 80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809 not found: ID does not exist"
Sep 16 20:53:19.534125 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.534091    2561 scope.go:117] "RemoveContainer" containerID="9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8"
Sep 16 20:53:19.534361 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.534342    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8"} err="failed to get container status \"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8\": rpc error: code = NotFound desc = could not find container \"9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8\": container with ID starting with 9e60f88cd14b8b785d0610c0f98847e6586480ebae5dbb99bff95caee3523df8 not found: ID does not exist"
Sep 16 20:53:19.534484 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.534362    2561 scope.go:117] "RemoveContainer" containerID="52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b"
Sep 16 20:53:19.534667 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.534647    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b"} err="failed to get container status \"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b\": rpc error: code = NotFound desc = could not find container \"52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b\": container with ID starting with 52387d412c45e65037692d0739af70462ee3d97651ffccb5ec34322ac4c1f60b not found: ID does not exist"
Sep 16 20:53:19.534734 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.534668    2561 scope.go:117] "RemoveContainer" containerID="80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809"
Sep 16 20:53:19.534902 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.534874    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809"} err="failed to get container status \"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809\": rpc error: code = NotFound desc = could not find container \"80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809\": container with ID starting with 80aeb050db4bdf02f42010d05102c7e698eda530cc016258971c24683a0d9809 not found: ID does not exist"
Sep 16 20:53:19.682912 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:19.682880    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" path="/var/lib/kubelet/pods/76cb49bf-1ec0-4666-b2fd-bf4acd4df10a/volumes"
Sep 16 20:53:28.170440 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:28.170399    2561 scope.go:117] "RemoveContainer" containerID="878cab20c26ebf59df33f8aeecbe31bdd4719c724116450ecf4cc93ff9e53476"
Sep 16 20:53:32.308989 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:32.308957    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"]
Sep 16 20:53:32.322224 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:32.322191    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"]
Sep 16 20:53:40.907716 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:40.907676    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:53:47.329718 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:47.329674    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerName="main" containerID="cri-o://d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee" gracePeriod=45
Sep 16 20:53:47.345660 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:47.345623    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main" containerID="cri-o://a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf" gracePeriod=45
Sep 16 20:53:47.961674 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:47.961624    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="llm-d-routing-sidecar" containerID="cri-o://8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22" gracePeriod=45
Sep 16 20:53:48.205083 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.205055    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:53:48.336506 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336471    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-model-cache\") pod \"f6b38b4c-25bc-4fad-95b2-12511f20562d\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") "
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336527    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6b38b4c-25bc-4fad-95b2-12511f20562d-cabundle-cert\") pod \"f6b38b4c-25bc-4fad-95b2-12511f20562d\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") "
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336554    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-tmp-dir\") pod \"f6b38b4c-25bc-4fad-95b2-12511f20562d\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") "
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336582    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-home\") pod \"f6b38b4c-25bc-4fad-95b2-12511f20562d\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") "
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336612    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-kserve-provision-location\") pod \"f6b38b4c-25bc-4fad-95b2-12511f20562d\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") "
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336639    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-dshm\") pod \"f6b38b4c-25bc-4fad-95b2-12511f20562d\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") "
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jn6v\" (UniqueName: \"kubernetes.io/projected/f6b38b4c-25bc-4fad-95b2-12511f20562d-kube-api-access-5jn6v\") pod \"f6b38b4c-25bc-4fad-95b2-12511f20562d\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") "
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b38b4c-25bc-4fad-95b2-12511f20562d-tls-certs\") pod \"f6b38b4c-25bc-4fad-95b2-12511f20562d\" (UID: \"f6b38b4c-25bc-4fad-95b2-12511f20562d\") "
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336771    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6b38b4c-25bc-4fad-95b2-12511f20562d" (UID: "f6b38b4c-25bc-4fad-95b2-12511f20562d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.336947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336940    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.337472 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.336980    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b38b4c-25bc-4fad-95b2-12511f20562d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f6b38b4c-25bc-4fad-95b2-12511f20562d" (UID: "f6b38b4c-25bc-4fad-95b2-12511f20562d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:53:48.337472 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.337173    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-home" (OuterVolumeSpecName: "home") pod "f6b38b4c-25bc-4fad-95b2-12511f20562d" (UID: "f6b38b4c-25bc-4fad-95b2-12511f20562d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.339042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.338995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-dshm" (OuterVolumeSpecName: "dshm") pod "f6b38b4c-25bc-4fad-95b2-12511f20562d" (UID: "f6b38b4c-25bc-4fad-95b2-12511f20562d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.339042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.339016    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b38b4c-25bc-4fad-95b2-12511f20562d-kube-api-access-5jn6v" (OuterVolumeSpecName: "kube-api-access-5jn6v") pod "f6b38b4c-25bc-4fad-95b2-12511f20562d" (UID: "f6b38b4c-25bc-4fad-95b2-12511f20562d"). InnerVolumeSpecName "kube-api-access-5jn6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:53:48.339289 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.339237    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b38b4c-25bc-4fad-95b2-12511f20562d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6b38b4c-25bc-4fad-95b2-12511f20562d" (UID: "f6b38b4c-25bc-4fad-95b2-12511f20562d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:53:48.351258 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.351232    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6b38b4c-25bc-4fad-95b2-12511f20562d" (UID: "f6b38b4c-25bc-4fad-95b2-12511f20562d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.359784 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.359756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6b38b4c-25bc-4fad-95b2-12511f20562d" (UID: "f6b38b4c-25bc-4fad-95b2-12511f20562d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.437709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.437635    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6b38b4c-25bc-4fad-95b2-12511f20562d-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.437709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.437663    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.437709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.437672    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.437709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.437682    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.437709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.437692    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b38b4c-25bc-4fad-95b2-12511f20562d-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.437709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.437701    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jn6v\" (UniqueName: \"kubernetes.io/projected/f6b38b4c-25bc-4fad-95b2-12511f20562d-kube-api-access-5jn6v\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.437709 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.437709    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b38b4c-25bc-4fad-95b2-12511f20562d-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.633437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.633398    2561 generic.go:356] "Generic (PLEG): container finished" podID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerID="a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf" exitCode=0
Sep 16 20:53:48.633437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.633431    2561 generic.go:356] "Generic (PLEG): container finished" podID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerID="8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22" exitCode=0
Sep 16 20:53:48.633624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.633464    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"
Sep 16 20:53:48.633624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.633475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" event={"ID":"f6b38b4c-25bc-4fad-95b2-12511f20562d","Type":"ContainerDied","Data":"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf"}
Sep 16 20:53:48.633624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.633533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" event={"ID":"f6b38b4c-25bc-4fad-95b2-12511f20562d","Type":"ContainerDied","Data":"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22"}
Sep 16 20:53:48.633624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.633561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6" event={"ID":"f6b38b4c-25bc-4fad-95b2-12511f20562d","Type":"ContainerDied","Data":"31e8bc3d3c96f4df038dd2b5eea5a7d888c6db01866aa8ae5f7c99b8b5eca655"}
Sep 16 20:53:48.633624 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.633592    2561 scope.go:117] "RemoveContainer" containerID="a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf"
Sep 16 20:53:48.641858 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.641835    2561 scope.go:117] "RemoveContainer" containerID="47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95"
Sep 16 20:53:48.649101 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.649081    2561 scope.go:117] "RemoveContainer" containerID="8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22"
Sep 16 20:53:48.656505 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.655771    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"]
Sep 16 20:53:48.656505 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.656208    2561 scope.go:117] "RemoveContainer" containerID="a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf"
Sep 16 20:53:48.656764 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:53:48.656742    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf\": container with ID starting with a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf not found: ID does not exist" containerID="a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf"
Sep 16 20:53:48.656830 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.656776    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf"} err="failed to get container status \"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf\": rpc error: code = NotFound desc = could not find container \"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf\": container with ID starting with a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf not found: ID does not exist"
Sep 16 20:53:48.656830 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.656801    2561 scope.go:117] "RemoveContainer" containerID="47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95"
Sep 16 20:53:48.657061 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:53:48.657042    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95\": container with ID starting with 47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95 not found: ID does not exist" containerID="47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95"
Sep 16 20:53:48.657105 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.657066    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95"} err="failed to get container status \"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95\": rpc error: code = NotFound desc = could not find container \"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95\": container with ID starting with 47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95 not found: ID does not exist"
Sep 16 20:53:48.657105 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.657082    2561 scope.go:117] "RemoveContainer" containerID="8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22"
Sep 16 20:53:48.657330 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:53:48.657314    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22\": container with ID starting with 8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22 not found: ID does not exist" containerID="8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22"
Sep 16 20:53:48.657417 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.657338    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22"} err="failed to get container status \"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22\": rpc error: code = NotFound desc = could not find container \"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22\": container with ID starting with 8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22 not found: ID does not exist"
Sep 16 20:53:48.657417 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.657359    2561 scope.go:117] "RemoveContainer" containerID="a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf"
Sep 16 20:53:48.657638 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.657618    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf"} err="failed to get container status \"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf\": rpc error: code = NotFound desc = could not find container \"a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf\": container with ID starting with a803e7b6cf8b2c891dd093926393d2af80b263f9f454e02658fe773f8abf70cf not found: ID does not exist"
Sep 16 20:53:48.657699 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.657639    2561 scope.go:117] "RemoveContainer" containerID="47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95"
Sep 16 20:53:48.657861 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.657841    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95"} err="failed to get container status \"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95\": rpc error: code = NotFound desc = could not find container \"47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95\": container with ID starting with 47d0ae66e4523bdeed69efea261672d82d0ca1c94c3b7a1d80157a8dc3ad5a95 not found: ID does not exist"
Sep 16 20:53:48.657916 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.657863    2561 scope.go:117] "RemoveContainer" containerID="8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22"
Sep 16 20:53:48.658066 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.658047    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22"} err="failed to get container status \"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22\": rpc error: code = NotFound desc = could not find container \"8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22\": container with ID starting with 8cd1bde4536cb9cbecf4fdebe6affe9974585f55f6e30a7727c1a8de1dce7f22 not found: ID does not exist"
Sep 16 20:53:48.660986 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:48.660963    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-84b6b568dc-nlhr6"]
Sep 16 20:53:49.681967 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:49.681936    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" path="/var/lib/kubelet/pods/f6b38b4c-25bc-4fad-95b2-12511f20562d/volumes"
Sep 16 20:53:50.907129 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:53:50.907090    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:54:00.907929 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:00.907890    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:54:00.908321 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:00.907988    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:54:10.907534 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:10.907441    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:54:13.959453 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:13.959420    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:54:13.966152 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:13.966130    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:54:17.573400 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.573358    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:54:17.656593 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.656497    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c636e4-b347-4e62-961b-06dcf397a69d-tls-certs\") pod \"a6c636e4-b347-4e62-961b-06dcf397a69d\" (UID: \"a6c636e4-b347-4e62-961b-06dcf397a69d\") "
Sep 16 20:54:17.656593 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.656546    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/a6c636e4-b347-4e62-961b-06dcf397a69d-kube-api-access-phwkn\") pod \"a6c636e4-b347-4e62-961b-06dcf397a69d\" (UID: \"a6c636e4-b347-4e62-961b-06dcf397a69d\") "
Sep 16 20:54:17.658670 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.658638    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c636e4-b347-4e62-961b-06dcf397a69d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6c636e4-b347-4e62-961b-06dcf397a69d" (UID: "a6c636e4-b347-4e62-961b-06dcf397a69d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:54:17.658785 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.658644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c636e4-b347-4e62-961b-06dcf397a69d-kube-api-access-phwkn" (OuterVolumeSpecName: "kube-api-access-phwkn") pod "a6c636e4-b347-4e62-961b-06dcf397a69d" (UID: "a6c636e4-b347-4e62-961b-06dcf397a69d"). InnerVolumeSpecName "kube-api-access-phwkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:54:17.749524 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.749491    2561 generic.go:356] "Generic (PLEG): container finished" podID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerID="d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee" exitCode=0
Sep 16 20:54:17.749689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.749533    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"
Sep 16 20:54:17.749689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.749571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" event={"ID":"a6c636e4-b347-4e62-961b-06dcf397a69d","Type":"ContainerDied","Data":"d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee"}
Sep 16 20:54:17.749689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.749642    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs" event={"ID":"a6c636e4-b347-4e62-961b-06dcf397a69d","Type":"ContainerDied","Data":"46500b6c41110268910f1314ba817d1127e596765b2d79d64e29ae96c3ce80fd"}
Sep 16 20:54:17.749689 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.749681    2561 scope.go:117] "RemoveContainer" containerID="d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee"
Sep 16 20:54:17.757623 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.757593    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c636e4-b347-4e62-961b-06dcf397a69d-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:54:17.757767 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.757626    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phwkn\" (UniqueName: \"kubernetes.io/projected/a6c636e4-b347-4e62-961b-06dcf397a69d-kube-api-access-phwkn\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:54:17.758289 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.758273    2561 scope.go:117] "RemoveContainer" containerID="d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee"
Sep 16 20:54:17.758573 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:54:17.758554    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee\": container with ID starting with d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee not found: ID does not exist" containerID="d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee"
Sep 16 20:54:17.758649 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.758584    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee"} err="failed to get container status \"d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee\": rpc error: code = NotFound desc = could not find container \"d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee\": container with ID starting with d45324081536f41ca770d827e3384710a561d610962904db7dc7023f6c4896ee not found: ID does not exist"
Sep 16 20:54:17.765884 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.765859    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"]
Sep 16 20:54:17.774201 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:17.769848    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586smhvs"]
Sep 16 20:54:19.681781 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:54:19.681743    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" path="/var/lib/kubelet/pods/a6c636e4-b347-4e62-961b-06dcf397a69d/volumes"
Sep 16 20:55:02.200089 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200045    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"]
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200449    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="llm-d-routing-sidecar"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200470    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="llm-d-routing-sidecar"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200484    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="storage-initializer"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200493    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="storage-initializer"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="llm-d-routing-sidecar"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200535    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="llm-d-routing-sidecar"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200544    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerName="main"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200553    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerName="main"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200567    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200576    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200589    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="storage-initializer"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200598    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="storage-initializer"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200617    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200626    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200698    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="main"
Sep 16 20:55:02.200702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200707    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="llm-d-routing-sidecar"
Sep 16 20:55:02.201525 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200716    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b38b4c-25bc-4fad-95b2-12511f20562d" containerName="main"
Sep 16 20:55:02.201525 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200724    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="76cb49bf-1ec0-4666-b2fd-bf4acd4df10a" containerName="llm-d-routing-sidecar"
Sep 16 20:55:02.201525 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.200732    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c636e4-b347-4e62-961b-06dcf397a69d" containerName="main"
Sep 16 20:55:02.203652 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.203627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.206403 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.206355    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 20:55:02.206545 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.206416    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 20:55:02.206545 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.206422    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 20:55:02.206545 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.206528    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:55:02.206721 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.206575    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 20:55:02.206721 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.206593    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-qz9dr\""
Sep 16 20:55:02.216927 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.216897    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"]
Sep 16 20:55:02.282679 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.282634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3091265b-6f48-491c-877b-b8ed5128550b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.282679 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.282679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.282915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.282713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3091265b-6f48-491c-877b-b8ed5128550b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.282915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.282787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.282915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.282823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.282915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.282842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55wb9\" (UniqueName: \"kubernetes.io/projected/3091265b-6f48-491c-877b-b8ed5128550b-kube-api-access-55wb9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.282915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.282870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.282915 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.282904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384024 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.383983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384024 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3091265b-6f48-491c-877b-b8ed5128550b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384196 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384196 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384272 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55wb9\" (UniqueName: \"kubernetes.io/projected/3091265b-6f48-491c-877b-b8ed5128550b-kube-api-access-55wb9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384272 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384358 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384358 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3091265b-6f48-491c-877b-b8ed5128550b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384497 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384564 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384811 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.384947 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.384929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3091265b-6f48-491c-877b-b8ed5128550b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.386577 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.386559    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.386985 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.386964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3091265b-6f48-491c-877b-b8ed5128550b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.393408 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.393366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55wb9\" (UniqueName: \"kubernetes.io/projected/3091265b-6f48-491c-877b-b8ed5128550b-kube-api-access-55wb9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.513549 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.513507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:02.798042 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.797953    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"]
Sep 16 20:55:02.802989 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:55:02.802949    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3091265b_6f48_491c_877b_b8ed5128550b.slice/crio-b84f4c81793b5d14ddd7259abcaba1ccc5e919cee155dfe0394b1c5691a1004e WatchSource:0}: Error finding container b84f4c81793b5d14ddd7259abcaba1ccc5e919cee155dfe0394b1c5691a1004e: Status 404 returned error can't find the container with id b84f4c81793b5d14ddd7259abcaba1ccc5e919cee155dfe0394b1c5691a1004e
Sep 16 20:55:02.923952 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.923904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" event={"ID":"3091265b-6f48-491c-877b-b8ed5128550b","Type":"ContainerStarted","Data":"0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406"}
Sep 16 20:55:02.924155 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:02.923978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" event={"ID":"3091265b-6f48-491c-877b-b8ed5128550b","Type":"ContainerStarted","Data":"b84f4c81793b5d14ddd7259abcaba1ccc5e919cee155dfe0394b1c5691a1004e"}
Sep 16 20:55:05.939426 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:05.939371    2561 generic.go:356] "Generic (PLEG): container finished" podID="3091265b-6f48-491c-877b-b8ed5128550b" containerID="0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406" exitCode=0
Sep 16 20:55:05.939805 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:05.939447    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" event={"ID":"3091265b-6f48-491c-877b-b8ed5128550b","Type":"ContainerDied","Data":"0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406"}
Sep 16 20:55:06.945712 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:06.945671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" event={"ID":"3091265b-6f48-491c-877b-b8ed5128550b","Type":"ContainerStarted","Data":"b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8"}
Sep 16 20:55:06.969741 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:06.969691    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podStartSLOduration=2.696122268 podStartE2EDuration="4.969674608s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.27355234s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:55:06.968212461 +0000 UTC m=+4553.740082872" watchObservedRunningTime="2026-09-16 20:55:06.969674608 +0000 UTC m=+4553.741545020"
Sep 16 20:55:07.513918 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:07.513881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:12.514731 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:12.514681    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:55:12.516060 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:12.516027    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:55:22.514437 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:22.514371    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:55:31.793270 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.793232    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"]
Sep 16 20:55:31.796930 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.796908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:31.800726 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.800702    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-2t9hg\""
Sep 16 20:55:31.800870 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.800702    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 20:55:31.800941 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.800714    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 20:55:31.800941 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.800716    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 20:55:31.800941 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.800764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-fglz4\""
Sep 16 20:55:31.806596 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.806574    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"]
Sep 16 20:55:31.930620 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.930580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74107d1-2219-45e9-b4de-631fa3c8679a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr\" (UID: \"c74107d1-2219-45e9-b4de-631fa3c8679a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:31.930800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:31.930633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2bf\" (UniqueName: \"kubernetes.io/projected/c74107d1-2219-45e9-b4de-631fa3c8679a-kube-api-access-7v2bf\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr\" (UID: \"c74107d1-2219-45e9-b4de-631fa3c8679a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:32.031108 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:32.031069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74107d1-2219-45e9-b4de-631fa3c8679a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr\" (UID: \"c74107d1-2219-45e9-b4de-631fa3c8679a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:32.031279 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:32.031123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2bf\" (UniqueName: \"kubernetes.io/projected/c74107d1-2219-45e9-b4de-631fa3c8679a-kube-api-access-7v2bf\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr\" (UID: \"c74107d1-2219-45e9-b4de-631fa3c8679a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:32.033764 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:32.033735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74107d1-2219-45e9-b4de-631fa3c8679a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr\" (UID: \"c74107d1-2219-45e9-b4de-631fa3c8679a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:32.038752 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:32.038727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2bf\" (UniqueName: \"kubernetes.io/projected/c74107d1-2219-45e9-b4de-631fa3c8679a-kube-api-access-7v2bf\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr\" (UID: \"c74107d1-2219-45e9-b4de-631fa3c8679a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:32.107925 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:32.107839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:32.384882 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:32.384800    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"]
Sep 16 20:55:32.388449 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:55:32.388404    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74107d1_2219_45e9_b4de_631fa3c8679a.slice/crio-8c451b8f5f56e375138f5b037e8f07d1d59a9bbab898e85c31c2f09775c5eb94 WatchSource:0}: Error finding container 8c451b8f5f56e375138f5b037e8f07d1d59a9bbab898e85c31c2f09775c5eb94: Status 404 returned error can't find the container with id 8c451b8f5f56e375138f5b037e8f07d1d59a9bbab898e85c31c2f09775c5eb94
Sep 16 20:55:32.514306 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:32.514245    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:55:33.059711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:33.059674    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" event={"ID":"c74107d1-2219-45e9-b4de-631fa3c8679a","Type":"ContainerStarted","Data":"f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba"}
Sep 16 20:55:33.060157 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:33.059731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" event={"ID":"c74107d1-2219-45e9-b4de-631fa3c8679a","Type":"ContainerStarted","Data":"8c451b8f5f56e375138f5b037e8f07d1d59a9bbab898e85c31c2f09775c5eb94"}
Sep 16 20:55:33.060157 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:33.059787    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:55:33.077510 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:33.077447    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" podStartSLOduration=2.077426719 podStartE2EDuration="2.077426719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:55:33.074951664 +0000 UTC m=+4579.846822074" watchObservedRunningTime="2026-09-16 20:55:33.077426719 +0000 UTC m=+4579.849297135"
Sep 16 20:55:42.514468 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:42.514350    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:55:52.514277 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:55:52.514224    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:56:02.515035 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:56:02.514985    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:56:04.066031 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:56:04.065998    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:56:12.513980 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:56:12.513931    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:56:22.514402 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:56:22.514343    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:56:32.514563 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:56:32.514519    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:56:42.514000 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:56:42.513946    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:56:52.513973 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:56:52.513926    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:57:02.524597 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:02.524553    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:57:02.532850 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:02.532806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:57:07.198983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:07.198901    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"]
Sep 16 20:57:22.220825 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:22.220755    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" containerID="cri-o://b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8" gracePeriod=45
Sep 16 20:57:22.525460 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:22.525423    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 16 20:57:22.974818 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:22.974790    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:57:23.003116 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-tmp-dir\") pod \"3091265b-6f48-491c-877b-b8ed5128550b\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") "
Sep 16 20:57:23.003299 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003130    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-dshm\") pod \"3091265b-6f48-491c-877b-b8ed5128550b\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") "
Sep 16 20:57:23.003299 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3091265b-6f48-491c-877b-b8ed5128550b-tls-certs\") pod \"3091265b-6f48-491c-877b-b8ed5128550b\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") "
Sep 16 20:57:23.003299 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003213    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-kserve-provision-location\") pod \"3091265b-6f48-491c-877b-b8ed5128550b\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") "
Sep 16 20:57:23.003299 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55wb9\" (UniqueName: \"kubernetes.io/projected/3091265b-6f48-491c-877b-b8ed5128550b-kube-api-access-55wb9\") pod \"3091265b-6f48-491c-877b-b8ed5128550b\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") "
Sep 16 20:57:23.003299 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3091265b-6f48-491c-877b-b8ed5128550b-cabundle-cert\") pod \"3091265b-6f48-491c-877b-b8ed5128550b\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") "
Sep 16 20:57:23.003582 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003308    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-model-cache\") pod \"3091265b-6f48-491c-877b-b8ed5128550b\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") "
Sep 16 20:57:23.003582 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-home\") pod \"3091265b-6f48-491c-877b-b8ed5128550b\" (UID: \"3091265b-6f48-491c-877b-b8ed5128550b\") "
Sep 16 20:57:23.004013 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.003981    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-model-cache" (OuterVolumeSpecName: "model-cache") pod "3091265b-6f48-491c-877b-b8ed5128550b" (UID: "3091265b-6f48-491c-877b-b8ed5128550b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:23.004182 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.004000    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3091265b-6f48-491c-877b-b8ed5128550b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3091265b-6f48-491c-877b-b8ed5128550b" (UID: "3091265b-6f48-491c-877b-b8ed5128550b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:57:23.005003 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.004979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-home" (OuterVolumeSpecName: "home") pod "3091265b-6f48-491c-877b-b8ed5128550b" (UID: "3091265b-6f48-491c-877b-b8ed5128550b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:23.006123 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.006082    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3091265b-6f48-491c-877b-b8ed5128550b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3091265b-6f48-491c-877b-b8ed5128550b" (UID: "3091265b-6f48-491c-877b-b8ed5128550b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:57:23.006123 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.006091    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-dshm" (OuterVolumeSpecName: "dshm") pod "3091265b-6f48-491c-877b-b8ed5128550b" (UID: "3091265b-6f48-491c-877b-b8ed5128550b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:23.006289 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.006134    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3091265b-6f48-491c-877b-b8ed5128550b-kube-api-access-55wb9" (OuterVolumeSpecName: "kube-api-access-55wb9") pod "3091265b-6f48-491c-877b-b8ed5128550b" (UID: "3091265b-6f48-491c-877b-b8ed5128550b"). InnerVolumeSpecName "kube-api-access-55wb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:57:23.019999 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.019972    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3091265b-6f48-491c-877b-b8ed5128550b" (UID: "3091265b-6f48-491c-877b-b8ed5128550b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:23.027793 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.027770    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3091265b-6f48-491c-877b-b8ed5128550b" (UID: "3091265b-6f48-491c-877b-b8ed5128550b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:23.104735 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.104651    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.104735 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.104684    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.104735 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.104693    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3091265b-6f48-491c-877b-b8ed5128550b-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.104735 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.104705    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-kserve-provision-location\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.104735 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.104714    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55wb9\" (UniqueName: \"kubernetes.io/projected/3091265b-6f48-491c-877b-b8ed5128550b-kube-api-access-55wb9\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.104735 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.104723    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3091265b-6f48-491c-877b-b8ed5128550b-cabundle-cert\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.104735 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.104731    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.104735 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.104739    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3091265b-6f48-491c-877b-b8ed5128550b-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.531778 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.531741    2561 generic.go:356] "Generic (PLEG): container finished" podID="3091265b-6f48-491c-877b-b8ed5128550b" containerID="b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8" exitCode=0
Sep 16 20:57:23.531778 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.531780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" event={"ID":"3091265b-6f48-491c-877b-b8ed5128550b","Type":"ContainerDied","Data":"b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8"}
Sep 16 20:57:23.532319 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.531798    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"
Sep 16 20:57:23.532319 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.531822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv" event={"ID":"3091265b-6f48-491c-877b-b8ed5128550b","Type":"ContainerDied","Data":"b84f4c81793b5d14ddd7259abcaba1ccc5e919cee155dfe0394b1c5691a1004e"}
Sep 16 20:57:23.532319 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.531843    2561 scope.go:117] "RemoveContainer" containerID="b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8"
Sep 16 20:57:23.540556 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.540533    2561 scope.go:117] "RemoveContainer" containerID="0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406"
Sep 16 20:57:23.547690 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.547675    2561 scope.go:117] "RemoveContainer" containerID="b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8"
Sep 16 20:57:23.547941 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:57:23.547923    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8\": container with ID starting with b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8 not found: ID does not exist" containerID="b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8"
Sep 16 20:57:23.547981 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.547951    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8"} err="failed to get container status \"b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8\": rpc error: code = NotFound desc = could not find container \"b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8\": container with ID starting with b652cac3732ec22d7aab1e7779b59d1a96b4be192e90d304f1f3345586bcd9e8 not found: ID does not exist"
Sep 16 20:57:23.547981 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.547968    2561 scope.go:117] "RemoveContainer" containerID="0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406"
Sep 16 20:57:23.548186 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:57:23.548170    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406\": container with ID starting with 0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406 not found: ID does not exist" containerID="0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406"
Sep 16 20:57:23.548223 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.548191    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406"} err="failed to get container status \"0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406\": rpc error: code = NotFound desc = could not find container \"0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406\": container with ID starting with 0f896beb22c41c8f8723f264d8f9f711234c596c3f79483d124ca53301422406 not found: ID does not exist"
Sep 16 20:57:23.553325 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.553303    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"]
Sep 16 20:57:23.557366 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.557344    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-777f8877fd8vxvv"]
Sep 16 20:57:23.683157 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:23.683123    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3091265b-6f48-491c-877b-b8ed5128550b" path="/var/lib/kubelet/pods/3091265b-6f48-491c-877b-b8ed5128550b/volumes"
Sep 16 20:57:37.683307 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.683274    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"]
Sep 16 20:57:37.683817 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.683609    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="storage-initializer"
Sep 16 20:57:37.683817 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.683629    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="storage-initializer"
Sep 16 20:57:37.683817 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.683642    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main"
Sep 16 20:57:37.683817 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.683651    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main"
Sep 16 20:57:37.683817 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.683727    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3091265b-6f48-491c-877b-b8ed5128550b" containerName="main"
Sep 16 20:57:37.686548 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.686531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.690390 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.690351    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-d8rcr\""
Sep 16 20:57:37.690523 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.690351    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:57:37.690523 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.690351    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:57:37.690523 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.690356    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:57:37.692909 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.692884    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"]
Sep 16 20:57:37.817496 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.817456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.817496 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.817498    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.817772 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.817519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26sqf\" (UniqueName: \"kubernetes.io/projected/10922c38-c8e9-40d6-ada0-ba5a7b637f36-kube-api-access-26sqf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.817772 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.817566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.817772 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.817599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.817772 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.817722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919052 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919244 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919244 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26sqf\" (UniqueName: \"kubernetes.io/projected/10922c38-c8e9-40d6-ada0-ba5a7b637f36-kube-api-access-26sqf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919244 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919244 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919244 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919529 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919591 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.919649 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.919600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.921494 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.921470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.921677 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.921660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.933029 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.933007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26sqf\" (UniqueName: \"kubernetes.io/projected/10922c38-c8e9-40d6-ada0-ba5a7b637f36-kube-api-access-26sqf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lx89j\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:37.997619 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:37.997528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:38.268527 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:38.268491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"]
Sep 16 20:57:38.272002 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:57:38.271973    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10922c38_c8e9_40d6_ada0_ba5a7b637f36.slice/crio-f9be996c0a2d54e9d503da4d3bd5327ddfd8dec347e8c541f675013078b1fc71 WatchSource:0}: Error finding container f9be996c0a2d54e9d503da4d3bd5327ddfd8dec347e8c541f675013078b1fc71: Status 404 returned error can't find the container with id f9be996c0a2d54e9d503da4d3bd5327ddfd8dec347e8c541f675013078b1fc71
Sep 16 20:57:38.595297 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:38.595198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j" event={"ID":"10922c38-c8e9-40d6-ada0-ba5a7b637f36","Type":"ContainerStarted","Data":"b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24"}
Sep 16 20:57:38.595297 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:38.595264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j" event={"ID":"10922c38-c8e9-40d6-ada0-ba5a7b637f36","Type":"ContainerStarted","Data":"f9be996c0a2d54e9d503da4d3bd5327ddfd8dec347e8c541f675013078b1fc71"}
Sep 16 20:57:38.611634 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:38.611581    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j" podStartSLOduration=1.611564747 podStartE2EDuration="1.611564747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:57:38.610899542 +0000 UTC m=+4705.382769944" watchObservedRunningTime="2026-09-16 20:57:38.611564747 +0000 UTC m=+4705.383435158"
Sep 16 20:57:38.997832 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:38.997799    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:47.997713 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:47.997681    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:48.010085 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:48.010059    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:57:48.653864 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:57:48.653828    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:58:14.002049 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:14.002007    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"]
Sep 16 20:58:14.065770 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:14.065724    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:58:24.065404 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:24.065339    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:58:25.149343 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.149308    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"]
Sep 16 20:58:25.159080 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:58:25.159025    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:58:25.159080 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:58:25.159080 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:58:25.159080 ip-10-0-136-120 kubenswrapper[2561]:  > containerID="b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24" cmd=["/bin/sleep","15"]
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.159091    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j" message="" err=<
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]:  >
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:58:25.159104    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j" podUID="10922c38-c8e9-40d6-ada0-ba5a7b637f36" containerName="main" containerID="cri-o://b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24"
Sep 16 20:58:25.159398 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.159151    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j" podUID="10922c38-c8e9-40d6-ada0-ba5a7b637f36" containerName="main" containerID="cri-o://b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24" gracePeriod=60
Sep 16 20:58:25.502771 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.502744    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:58:25.591728 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.591691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-dshm\") pod \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") "
Sep 16 20:58:25.591983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.591749    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tmp-dir\") pod \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") "
Sep 16 20:58:25.591983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.591772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26sqf\" (UniqueName: \"kubernetes.io/projected/10922c38-c8e9-40d6-ada0-ba5a7b637f36-kube-api-access-26sqf\") pod \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") "
Sep 16 20:58:25.591983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.591801    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-home\") pod \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") "
Sep 16 20:58:25.591983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.591824    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tls-certs\") pod \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") "
Sep 16 20:58:25.591983 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.591851    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-model-cache\") pod \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\" (UID: \"10922c38-c8e9-40d6-ada0-ba5a7b637f36\") "
Sep 16 20:58:25.592260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.592104    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-model-cache" (OuterVolumeSpecName: "model-cache") pod "10922c38-c8e9-40d6-ada0-ba5a7b637f36" (UID: "10922c38-c8e9-40d6-ada0-ba5a7b637f36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:25.592260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.592114    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-home" (OuterVolumeSpecName: "home") pod "10922c38-c8e9-40d6-ada0-ba5a7b637f36" (UID: "10922c38-c8e9-40d6-ada0-ba5a7b637f36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:25.592260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.592135    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10922c38-c8e9-40d6-ada0-ba5a7b637f36" (UID: "10922c38-c8e9-40d6-ada0-ba5a7b637f36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:25.592260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.592235    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tmp-dir\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.592260 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.592252    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-home\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.592494 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.592265    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-model-cache\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.594064 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.594043    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-dshm" (OuterVolumeSpecName: "dshm") pod "10922c38-c8e9-40d6-ada0-ba5a7b637f36" (UID: "10922c38-c8e9-40d6-ada0-ba5a7b637f36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:25.594153 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.594104    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10922c38-c8e9-40d6-ada0-ba5a7b637f36-kube-api-access-26sqf" (OuterVolumeSpecName: "kube-api-access-26sqf") pod "10922c38-c8e9-40d6-ada0-ba5a7b637f36" (UID: "10922c38-c8e9-40d6-ada0-ba5a7b637f36"). InnerVolumeSpecName "kube-api-access-26sqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:58:25.594153 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.594117    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10922c38-c8e9-40d6-ada0-ba5a7b637f36" (UID: "10922c38-c8e9-40d6-ada0-ba5a7b637f36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:58:25.692711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.692685    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10922c38-c8e9-40d6-ada0-ba5a7b637f36-dshm\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.692711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.692709    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26sqf\" (UniqueName: \"kubernetes.io/projected/10922c38-c8e9-40d6-ada0-ba5a7b637f36-kube-api-access-26sqf\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.692711 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.692718    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10922c38-c8e9-40d6-ada0-ba5a7b637f36-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.796259 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.796215    2561 generic.go:356] "Generic (PLEG): container finished" podID="10922c38-c8e9-40d6-ada0-ba5a7b637f36" containerID="b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24" exitCode=0
Sep 16 20:58:25.796493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.796288    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"
Sep 16 20:58:25.796493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.796301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j" event={"ID":"10922c38-c8e9-40d6-ada0-ba5a7b637f36","Type":"ContainerDied","Data":"b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24"}
Sep 16 20:58:25.796493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.796407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j" event={"ID":"10922c38-c8e9-40d6-ada0-ba5a7b637f36","Type":"ContainerDied","Data":"f9be996c0a2d54e9d503da4d3bd5327ddfd8dec347e8c541f675013078b1fc71"}
Sep 16 20:58:25.796493 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.796433    2561 scope.go:117] "RemoveContainer" containerID="b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24"
Sep 16 20:58:25.805094 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.805063    2561 scope.go:117] "RemoveContainer" containerID="b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24"
Sep 16 20:58:25.805354 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:58:25.805335    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24\": container with ID starting with b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24 not found: ID does not exist" containerID="b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24"
Sep 16 20:58:25.805428 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.805361    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24"} err="failed to get container status \"b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24\": rpc error: code = NotFound desc = could not find container \"b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24\": container with ID starting with b09a382ce4af14d6b9e11382defeb05112e84953048d8f85664dc3584e678c24 not found: ID does not exist"
Sep 16 20:58:25.814441 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.814415    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"]
Sep 16 20:58:25.817988 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:25.817959    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lx89j"]
Sep 16 20:58:27.682395 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:27.682353    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10922c38-c8e9-40d6-ada0-ba5a7b637f36" path="/var/lib/kubelet/pods/10922c38-c8e9-40d6-ada0-ba5a7b637f36/volumes"
Sep 16 20:58:29.026918 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:29.026872    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main" containerID="cri-o://f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba" gracePeriod=45
Sep 16 20:58:34.065956 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:34.065910    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:58:34.066428 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:34.066027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:58:44.066446 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:44.066340    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:58:54.066361 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:54.066321    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:58:59.268075 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.268045    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:58:59.315309 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.315229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74107d1-2219-45e9-b4de-631fa3c8679a-tls-certs\") pod \"c74107d1-2219-45e9-b4de-631fa3c8679a\" (UID: \"c74107d1-2219-45e9-b4de-631fa3c8679a\") "
Sep 16 20:58:59.315309 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.315296    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v2bf\" (UniqueName: \"kubernetes.io/projected/c74107d1-2219-45e9-b4de-631fa3c8679a-kube-api-access-7v2bf\") pod \"c74107d1-2219-45e9-b4de-631fa3c8679a\" (UID: \"c74107d1-2219-45e9-b4de-631fa3c8679a\") "
Sep 16 20:58:59.317519 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.317487    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74107d1-2219-45e9-b4de-631fa3c8679a-kube-api-access-7v2bf" (OuterVolumeSpecName: "kube-api-access-7v2bf") pod "c74107d1-2219-45e9-b4de-631fa3c8679a" (UID: "c74107d1-2219-45e9-b4de-631fa3c8679a"). InnerVolumeSpecName "kube-api-access-7v2bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:58:59.317637 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.317528    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74107d1-2219-45e9-b4de-631fa3c8679a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c74107d1-2219-45e9-b4de-631fa3c8679a" (UID: "c74107d1-2219-45e9-b4de-631fa3c8679a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:58:59.415910 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.415862    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74107d1-2219-45e9-b4de-631fa3c8679a-tls-certs\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:58:59.415910 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.415906    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v2bf\" (UniqueName: \"kubernetes.io/projected/c74107d1-2219-45e9-b4de-631fa3c8679a-kube-api-access-7v2bf\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:58:59.939259 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.939223    2561 generic.go:356] "Generic (PLEG): container finished" podID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerID="f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba" exitCode=0
Sep 16 20:58:59.939497 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.939264    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"
Sep 16 20:58:59.939497 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.939277    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" event={"ID":"c74107d1-2219-45e9-b4de-631fa3c8679a","Type":"ContainerDied","Data":"f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba"}
Sep 16 20:58:59.939497 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.939323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr" event={"ID":"c74107d1-2219-45e9-b4de-631fa3c8679a","Type":"ContainerDied","Data":"8c451b8f5f56e375138f5b037e8f07d1d59a9bbab898e85c31c2f09775c5eb94"}
Sep 16 20:58:59.939497 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.939346    2561 scope.go:117] "RemoveContainer" containerID="f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba"
Sep 16 20:58:59.947551 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.947530    2561 scope.go:117] "RemoveContainer" containerID="f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba"
Sep 16 20:58:59.947807 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:58:59.947779    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba\": container with ID starting with f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba not found: ID does not exist" containerID="f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba"
Sep 16 20:58:59.947860 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.947816    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba"} err="failed to get container status \"f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba\": rpc error: code = NotFound desc = could not find container \"f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba\": container with ID starting with f9c0ad3dabebb32098fb86f91b4ecbd8627a36ad2af2258ae0b62339d65405ba not found: ID does not exist"
Sep 16 20:58:59.957087 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.957063    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"]
Sep 16 20:58:59.960742 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:58:59.960722    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-n7llr"]
Sep 16 20:59:01.682308 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:01.682268    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" path="/var/lib/kubelet/pods/c74107d1-2219-45e9-b4de-631fa3c8679a/volumes"
Sep 16 20:59:13.981800 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:13.981765    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:59:13.989840 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:13.989814    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhw67_bbcd8fa0-2baf-4439-961f-b629b3a1b899/ovn-acl-logging/0.log"
Sep 16 20:59:16.439287 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.439251    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzphm/must-gather-54jb4"]
Sep 16 20:59:16.439702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.439542    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main"
Sep 16 20:59:16.439702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.439556    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main"
Sep 16 20:59:16.439702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.439566    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10922c38-c8e9-40d6-ada0-ba5a7b637f36" containerName="main"
Sep 16 20:59:16.439702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.439572    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="10922c38-c8e9-40d6-ada0-ba5a7b637f36" containerName="main"
Sep 16 20:59:16.439702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.439638    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74107d1-2219-45e9-b4de-631fa3c8679a" containerName="main"
Sep 16 20:59:16.439702 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.439645    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="10922c38-c8e9-40d6-ada0-ba5a7b637f36" containerName="main"
Sep 16 20:59:16.442625 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.442603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:16.445528 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.445490    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzphm\"/\"kube-root-ca.crt\""
Sep 16 20:59:16.445648 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.445577    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzphm\"/\"openshift-service-ca.crt\""
Sep 16 20:59:16.445727 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.445652    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zzphm\"/\"default-dockercfg-9bkbx\""
Sep 16 20:59:16.452533 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.452499    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzphm/must-gather-54jb4"]
Sep 16 20:59:16.546325 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.546287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z444w\" (UniqueName: \"kubernetes.io/projected/96400ee2-0d9e-49cd-9715-03345f04e875-kube-api-access-z444w\") pod \"must-gather-54jb4\" (UID: \"96400ee2-0d9e-49cd-9715-03345f04e875\") " pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:16.546325 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.546332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96400ee2-0d9e-49cd-9715-03345f04e875-must-gather-output\") pod \"must-gather-54jb4\" (UID: \"96400ee2-0d9e-49cd-9715-03345f04e875\") " pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:16.646747 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.646709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z444w\" (UniqueName: \"kubernetes.io/projected/96400ee2-0d9e-49cd-9715-03345f04e875-kube-api-access-z444w\") pod \"must-gather-54jb4\" (UID: \"96400ee2-0d9e-49cd-9715-03345f04e875\") " pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:16.646747 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.646752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96400ee2-0d9e-49cd-9715-03345f04e875-must-gather-output\") pod \"must-gather-54jb4\" (UID: \"96400ee2-0d9e-49cd-9715-03345f04e875\") " pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:16.647065 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.647049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96400ee2-0d9e-49cd-9715-03345f04e875-must-gather-output\") pod \"must-gather-54jb4\" (UID: \"96400ee2-0d9e-49cd-9715-03345f04e875\") " pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:16.660753 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.660719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z444w\" (UniqueName: \"kubernetes.io/projected/96400ee2-0d9e-49cd-9715-03345f04e875-kube-api-access-z444w\") pod \"must-gather-54jb4\" (UID: \"96400ee2-0d9e-49cd-9715-03345f04e875\") " pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:16.752195 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:16.752164    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:17.045275 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:17.045249    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzphm/must-gather-54jb4"]
Sep 16 20:59:17.047828 ip-10-0-136-120 kubenswrapper[2561]: W0916 20:59:17.047799    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96400ee2_0d9e_49cd_9715_03345f04e875.slice/crio-7026837d8032d6f6c283f1fdfafbf18d3c558a14012c1c15623e46a68856a7ef WatchSource:0}: Error finding container 7026837d8032d6f6c283f1fdfafbf18d3c558a14012c1c15623e46a68856a7ef: Status 404 returned error can't find the container with id 7026837d8032d6f6c283f1fdfafbf18d3c558a14012c1c15623e46a68856a7ef
Sep 16 20:59:17.049634 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:17.049616    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:59:18.020161 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:18.020119    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzphm/must-gather-54jb4" event={"ID":"96400ee2-0d9e-49cd-9715-03345f04e875","Type":"ContainerStarted","Data":"7026837d8032d6f6c283f1fdfafbf18d3c558a14012c1c15623e46a68856a7ef"}
Sep 16 20:59:24.055366 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:24.055330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzphm/must-gather-54jb4" event={"ID":"96400ee2-0d9e-49cd-9715-03345f04e875","Type":"ContainerStarted","Data":"fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b"}
Sep 16 20:59:24.055810 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:24.055400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzphm/must-gather-54jb4" event={"ID":"96400ee2-0d9e-49cd-9715-03345f04e875","Type":"ContainerStarted","Data":"7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67"}
Sep 16 20:59:24.072360 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:24.072309    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzphm/must-gather-54jb4" podStartSLOduration=1.8239858450000002 podStartE2EDuration="8.072294521s" totalImagesPullingTime="6.248308676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:59:24.071534552 +0000 UTC m=+4810.843404958" watchObservedRunningTime="2026-09-16 20:59:24.072294521 +0000 UTC m=+4810.844164931"
Sep 16 20:59:41.845050 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:41.845022    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9kf4f_fa4d18f8-2eb3-4ad0-9448-7b601d74aeab/kuadrant-console-plugin/0.log"
Sep 16 20:59:47.158044 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:47.158006    2561 generic.go:356] "Generic (PLEG): container finished" podID="96400ee2-0d9e-49cd-9715-03345f04e875" containerID="7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67" exitCode=0
Sep 16 20:59:47.158491 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:47.158065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzphm/must-gather-54jb4" event={"ID":"96400ee2-0d9e-49cd-9715-03345f04e875","Type":"ContainerDied","Data":"7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67"}
Sep 16 20:59:47.158491 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:47.158426    2561 scope.go:117] "RemoveContainer" containerID="7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67"
Sep 16 20:59:47.464530 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:47.464445    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzphm_must-gather-54jb4_96400ee2-0d9e-49cd-9715-03345f04e875/gather/0.log"
Sep 16 20:59:51.176757 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:51.176723    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wct9n_02086a7b-8320-4264-96c3-fba73847dbce/global-pull-secret-syncer/0.log"
Sep 16 20:59:51.317095 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:51.317066    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z26db_07142bb7-f90d-4bb3-b051-671f06731298/konnectivity-agent/0.log"
Sep 16 20:59:51.338446 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:51.338416    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-120.ec2.internal_1e2c558c42ad344dfa74981a35dcbdcb/haproxy/0.log"
Sep 16 20:59:52.922896 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:52.922855    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zzphm/must-gather-54jb4"]
Sep 16 20:59:52.923361 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:52.923125    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-zzphm/must-gather-54jb4" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" containerName="copy" containerID="cri-o://fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b" gracePeriod=2
Sep 16 20:59:52.929892 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:52.929831    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zzphm/must-gather-54jb4"]
Sep 16 20:59:52.931411 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:59:52.931353    2561 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"96400ee2-0d9e-49cd-9715-03345f04e875\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-zzphm\"/\"must-gather-54jb4\": pods \"must-gather-54jb4\" not found" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:53.158187 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.158161    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzphm_must-gather-54jb4_96400ee2-0d9e-49cd-9715-03345f04e875/copy/0.log"
Sep 16 20:59:53.159576 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.159558    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:53.162681 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:59:53.162658    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-54jb4\" is forbidden: User \"system:node:ip-10-0-136-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzphm\": no relationship found between node 'ip-10-0-136-120.ec2.internal' and this object" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:53.182306 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.182248    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzphm_must-gather-54jb4_96400ee2-0d9e-49cd-9715-03345f04e875/copy/0.log"
Sep 16 20:59:53.182646 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.182626    2561 generic.go:356] "Generic (PLEG): container finished" podID="96400ee2-0d9e-49cd-9715-03345f04e875" containerID="fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b" exitCode=143
Sep 16 20:59:53.182722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.182670    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:53.182722 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.182681    2561 scope.go:117] "RemoveContainer" containerID="fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b"
Sep 16 20:59:53.185081 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:59:53.185059    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-54jb4\" is forbidden: User \"system:node:ip-10-0-136-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzphm\": no relationship found between node 'ip-10-0-136-120.ec2.internal' and this object" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:53.190177 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.190145    2561 scope.go:117] "RemoveContainer" containerID="7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67"
Sep 16 20:59:53.203591 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.203571    2561 scope.go:117] "RemoveContainer" containerID="fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b"
Sep 16 20:59:53.203859 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:59:53.203836    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b\": container with ID starting with fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b not found: ID does not exist" containerID="fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b"
Sep 16 20:59:53.203922 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.203868    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b"} err="failed to get container status \"fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b\": rpc error: code = NotFound desc = could not find container \"fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b\": container with ID starting with fe3054f19fda2328bad1b1977509ee107d2b1378aac5a081131e7aff3e68115b not found: ID does not exist"
Sep 16 20:59:53.203922 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.203886    2561 scope.go:117] "RemoveContainer" containerID="7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67"
Sep 16 20:59:53.204141 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:59:53.204123    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67\": container with ID starting with 7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67 not found: ID does not exist" containerID="7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67"
Sep 16 20:59:53.204182 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.204149    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67"} err="failed to get container status \"7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67\": rpc error: code = NotFound desc = could not find container \"7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67\": container with ID starting with 7f0a21bb3c9900dae35a1da421bcc0aff8db97cfe1f4bdf7e2e4b86ce6c0bc67 not found: ID does not exist"
Sep 16 20:59:53.245442 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.245413    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96400ee2-0d9e-49cd-9715-03345f04e875-must-gather-output\") pod \"96400ee2-0d9e-49cd-9715-03345f04e875\" (UID: \"96400ee2-0d9e-49cd-9715-03345f04e875\") "
Sep 16 20:59:53.245541 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.245455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z444w\" (UniqueName: \"kubernetes.io/projected/96400ee2-0d9e-49cd-9715-03345f04e875-kube-api-access-z444w\") pod \"96400ee2-0d9e-49cd-9715-03345f04e875\" (UID: \"96400ee2-0d9e-49cd-9715-03345f04e875\") "
Sep 16 20:59:53.247826 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.247802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96400ee2-0d9e-49cd-9715-03345f04e875-kube-api-access-z444w" (OuterVolumeSpecName: "kube-api-access-z444w") pod "96400ee2-0d9e-49cd-9715-03345f04e875" (UID: "96400ee2-0d9e-49cd-9715-03345f04e875"). InnerVolumeSpecName "kube-api-access-z444w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:59:53.252770 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.252745    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96400ee2-0d9e-49cd-9715-03345f04e875-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "96400ee2-0d9e-49cd-9715-03345f04e875" (UID: "96400ee2-0d9e-49cd-9715-03345f04e875"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:59:53.346348 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.346309    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96400ee2-0d9e-49cd-9715-03345f04e875-must-gather-output\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:59:53.346348 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.346340    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z444w\" (UniqueName: \"kubernetes.io/projected/96400ee2-0d9e-49cd-9715-03345f04e875-kube-api-access-z444w\") on node \"ip-10-0-136-120.ec2.internal\" DevicePath \"\""
Sep 16 20:59:53.494029 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:59:53.493990    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-54jb4\" is forbidden: User \"system:node:ip-10-0-136-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzphm\": no relationship found between node 'ip-10-0-136-120.ec2.internal' and this object" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:53.681344 ip-10-0-136-120 kubenswrapper[2561]: E0916 20:59:53.681303    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-54jb4\" is forbidden: User \"system:node:ip-10-0-136-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzphm\": no relationship found between node 'ip-10-0-136-120.ec2.internal' and this object" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" pod="openshift-must-gather-zzphm/must-gather-54jb4"
Sep 16 20:59:53.682656 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:53.682624    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" path="/var/lib/kubelet/pods/96400ee2-0d9e-49cd-9715-03345f04e875/volumes"
Sep 16 20:59:56.835783 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:56.835755    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9kf4f_fa4d18f8-2eb3-4ad0-9448-7b601d74aeab/kuadrant-console-plugin/0.log"
Sep 16 20:59:58.736070 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:58.735996    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm7d8_67c8c80b-22e6-4698-9a1d-b9f25ca9a089/node-exporter/0.log"
Sep 16 20:59:58.754407 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:58.754385    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm7d8_67c8c80b-22e6-4698-9a1d-b9f25ca9a089/kube-rbac-proxy/0.log"
Sep 16 20:59:58.772613 ip-10-0-136-120 kubenswrapper[2561]: I0916 20:59:58.772591    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm7d8_67c8c80b-22e6-4698-9a1d-b9f25ca9a089/init-textfile/0.log"
Sep 16 21:00:01.194105 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.194067    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"]
Sep 16 21:00:01.194511 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.194321    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" containerName="copy"
Sep 16 21:00:01.194511 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.194332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" containerName="copy"
Sep 16 21:00:01.194511 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.194350    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" containerName="gather"
Sep 16 21:00:01.194511 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.194355    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" containerName="gather"
Sep 16 21:00:01.194511 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.194411    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" containerName="copy"
Sep 16 21:00:01.194511 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.194421    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="96400ee2-0d9e-49cd-9715-03345f04e875" containerName="gather"
Sep 16 21:00:01.198260 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.198240    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.200871 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.200842    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9wkzj\"/\"default-dockercfg-qhzj8\""
Sep 16 21:00:01.202082 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.202057    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9wkzj\"/\"kube-root-ca.crt\""
Sep 16 21:00:01.202157 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.202069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9wkzj\"/\"openshift-service-ca.crt\""
Sep 16 21:00:01.207405 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.207366    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"]
Sep 16 21:00:01.303556 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.303513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-proc\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.303761 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.303568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2n25\" (UniqueName: \"kubernetes.io/projected/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-kube-api-access-b2n25\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.303761 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.303600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-podres\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.303761 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.303635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-sys\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.303761 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.303703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-lib-modules\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404475 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-proc\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404654 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2n25\" (UniqueName: \"kubernetes.io/projected/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-kube-api-access-b2n25\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404654 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-podres\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404654 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-sys\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404654 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-proc\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404654 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-lib-modules\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404654 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-sys\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404865 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404689    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-podres\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.404865 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.404767    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-lib-modules\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.412937 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.412909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2n25\" (UniqueName: \"kubernetes.io/projected/2fd609ff-0cb8-4cb9-9e4d-151aa3107240-kube-api-access-b2n25\") pod \"perf-node-gather-daemonset-fv6kc\" (UID: \"2fd609ff-0cb8-4cb9-9e4d-151aa3107240\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.508524 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.508491    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:01.778651 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:01.778555    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"]
Sep 16 21:00:01.781977 ip-10-0-136-120 kubenswrapper[2561]: W0916 21:00:01.781952    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2fd609ff_0cb8_4cb9_9e4d_151aa3107240.slice/crio-e838a81a38917f7958bcb84c2073b3591385b8dc82064f8e23e90b45d89e8daa WatchSource:0}: Error finding container e838a81a38917f7958bcb84c2073b3591385b8dc82064f8e23e90b45d89e8daa: Status 404 returned error can't find the container with id e838a81a38917f7958bcb84c2073b3591385b8dc82064f8e23e90b45d89e8daa
Sep 16 21:00:02.220665 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:02.220566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc" event={"ID":"2fd609ff-0cb8-4cb9-9e4d-151aa3107240","Type":"ContainerStarted","Data":"3e1df8860d2840b5064a80af0a3abc8e63a5b6258a0a9e3ca99fed0e3dc1512f"}
Sep 16 21:00:02.220665 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:02.220632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc" event={"ID":"2fd609ff-0cb8-4cb9-9e4d-151aa3107240","Type":"ContainerStarted","Data":"e838a81a38917f7958bcb84c2073b3591385b8dc82064f8e23e90b45d89e8daa"}
Sep 16 21:00:02.221072 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:02.220679    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:02.238475 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:02.238416    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc" podStartSLOduration=1.238399367 podStartE2EDuration="1.238399367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:00:02.237272331 +0000 UTC m=+4849.009142742" watchObservedRunningTime="2026-09-16 21:00:02.238399367 +0000 UTC m=+4849.010269792"
Sep 16 21:00:03.196097 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:03.196068    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ndpfg_c42e50fb-1452-4a79-bf3d-ccaf42264113/dns/0.log"
Sep 16 21:00:03.216477 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:03.216448    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ndpfg_c42e50fb-1452-4a79-bf3d-ccaf42264113/kube-rbac-proxy/0.log"
Sep 16 21:00:03.266971 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:03.266934    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fjkh7_bed5bc13-28aa-4988-a9ca-0bc86e03637e/dns-node-resolver/0.log"
Sep 16 21:00:03.850224 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:03.850190    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5l96d_7529a9e4-1769-414e-939e-79eae5b17d12/node-ca/0.log"
Sep 16 21:00:05.578867 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:05.578839    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn4vj_cc075fa1-4b85-4a98-ab9b-2c0a62d4bfaa/serve-healthcheck-canary/0.log"
Sep 16 21:00:06.229172 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:06.229094    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnzbm_f517ca1a-8142-48df-83f2-f442fcd61215/kube-rbac-proxy/0.log"
Sep 16 21:00:06.257129 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:06.257075    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnzbm_f517ca1a-8142-48df-83f2-f442fcd61215/exporter/0.log"
Sep 16 21:00:06.279655 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:06.279621    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnzbm_f517ca1a-8142-48df-83f2-f442fcd61215/extractor/0.log"
Sep 16 21:00:08.234213 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:08.234182    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-fv6kc"
Sep 16 21:00:09.324263 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:09.324234    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b4645d7c8-gvj24_965e977c-2a61-49ae-9d6b-0e071efcc307/manager/0.log"
Sep 16 21:00:10.070352 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:10.070323    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7f95f9f864-c7kdq_eeedf724-46ab-42f8-800e-c88916f4f513/manager/0.log"
Sep 16 21:00:10.430607 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:10.430572    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-668c6bff64-76km2_b034f221-c36c-4dda-90fb-5bb402d985ae/manager/0.log"
Sep 16 21:00:12.260968 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:12.260941    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-p8mb7_7a4c3429-f98b-4971-b3d2-253c66134b42/s3-init/0.log"
Sep 16 21:00:12.276229 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:12.276205    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-p8mb7_7a4c3429-f98b-4971-b3d2-253c66134b42/download-hf-model/0.log"
Sep 16 21:00:18.347079 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.347044    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8znj5_0e8ea2e8-1fbc-4e09-97b7-f2b56caa7131/kube-multus/0.log"
Sep 16 21:00:18.373566 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.373541    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-525g9_15ecb2f2-b01c-4588-b82a-a8f6bde8d89c/kube-multus-additional-cni-plugins/0.log"
Sep 16 21:00:18.397189 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.397163    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-525g9_15ecb2f2-b01c-4588-b82a-a8f6bde8d89c/egress-router-binary-copy/0.log"
Sep 16 21:00:18.418583 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.418563    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-525g9_15ecb2f2-b01c-4588-b82a-a8f6bde8d89c/cni-plugins/0.log"
Sep 16 21:00:18.442017 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.441998    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-525g9_15ecb2f2-b01c-4588-b82a-a8f6bde8d89c/bond-cni-plugin/0.log"
Sep 16 21:00:18.467984 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.467964    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-525g9_15ecb2f2-b01c-4588-b82a-a8f6bde8d89c/routeoverride-cni/0.log"
Sep 16 21:00:18.497493 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.497471    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-525g9_15ecb2f2-b01c-4588-b82a-a8f6bde8d89c/whereabouts-cni-bincopy/0.log"
Sep 16 21:00:18.521941 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.521920    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-525g9_15ecb2f2-b01c-4588-b82a-a8f6bde8d89c/whereabouts-cni/0.log"
Sep 16 21:00:18.946828 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.946797    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cc5h9_ca3b10ac-d514-413b-aa81-23fbe19df3bd/network-metrics-daemon/0.log"
Sep 16 21:00:18.965784 ip-10-0-136-120 kubenswrapper[2561]: I0916 21:00:18.965760    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cc5h9_ca3b10ac-d514-413b-aa81-23fbe19df3bd/kube-rbac-proxy/0.log"