Jul 20 09:54:52.358519 ip-10-0-138-196 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 20 09:54:52.358531 ip-10-0-138-196 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 20 09:54:52.358538 ip-10-0-138-196 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 20 09:54:52.358746 ip-10-0-138-196 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 20 09:55:02.605895 ip-10-0-138-196 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 20 09:55:02.605914 ip-10-0-138-196 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ef5ea24531594ff78400d29b59ee3195 --
Jul 20 09:56:58.475619 ip-10-0-138-196 systemd[1]: Starting Kubernetes Kubelet...
Jul 20 09:56:58.873954 ip-10-0-138-196 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:58.873954 ip-10-0-138-196 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 20 09:56:58.873954 ip-10-0-138-196 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:58.873954 ip-10-0-138-196 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 20 09:56:58.873954 ip-10-0-138-196 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:58.874854 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.874736    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 20 09:56:58.878291 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878275    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:58.878291 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878292    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878296    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878299    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878302    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878308    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878312    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878315    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878318    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878321    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878324    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878327    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878330    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878344    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878347    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878350    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878352    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878355    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878358    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878361    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:58.878376 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878364    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878366    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878369    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878379    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878382    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878385    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878388    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878390    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878393    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878397    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878400    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878402    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878404    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878407    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878410    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878412    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878416    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878419    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878423    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878426    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:58.878833 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878429    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878431    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878434    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878438    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878442    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878445    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878447    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878450    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878453    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878455    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878458    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878460    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878463    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878466    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878470    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878473    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878475    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878478    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878480    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878483    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:58.879301 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878486    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878488    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878491    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878493    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878495    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878498    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878500    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878503    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878506    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878508    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878511    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878514    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878516    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878519    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878521    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878524    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878526    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878529    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878531    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:58.879812 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878533    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878536    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878538    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878541    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878543    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878545    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878548    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878551    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878554    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878556    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878559    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878562    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878564    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878566    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878569    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878571    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878574    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878576    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878579    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.878581    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:58.880252 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879266    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879273    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879275    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879278    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879281    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879284    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879286    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879289    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879291    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879294    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879296    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879299    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879303    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879307    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879311    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879314    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879316    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879319    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879322    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:58.880752 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879326    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879329    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879348    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879351    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879354    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879357    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879359    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879362    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879364    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879367    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879369    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879372    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879376    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879379    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879382    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879385    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879388    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879391    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879394    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:58.881199 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879397    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879400    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879402    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879404    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879407    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879410    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879412    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879414    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879418    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879421    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879424    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879426    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879428    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879433    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879435    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879438    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879440    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879443    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879446    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:58.881687 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879448    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879450    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879453    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879455    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879458    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879461    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879463    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879466    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879468    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879471    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879474    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879476    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879479    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879481    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879484    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879487    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879489    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879492    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879494    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879497    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:58.882144 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879499    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879503    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879505    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879508    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879510    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879513    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879516    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879519    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879522    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879524    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879527    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879530    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879532    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879535    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879538    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879541    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879543    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879546    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879548    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879551    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:58.882638 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879553    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879555    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879921    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879926    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879929    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879932    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879935    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879938    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879941    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879943    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879946    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879948    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879951    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879953    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879956    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879958    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879961    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879964    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879967    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879970    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:58.883109 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879972    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879975    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879977    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879980    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879983    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879985    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879988    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879990    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879993    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879995    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.879998    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880000    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880003    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880005    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880009    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880013    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880016    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880019    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880022    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880025    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:58.883614 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880028    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880031    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880034    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880036    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880039    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880043    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880046    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880049    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880052    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880054    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880058    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880061    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880063    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880066    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880068    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880070    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880074    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880076    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880079    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:58.884091 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880081    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880083    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880086    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880088    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880091    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880094    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880096    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880098    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880101    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880103    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880106    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880108    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880110    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880113    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880116    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880118    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880121    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880123    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880126    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880128    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880130    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:58.884557 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880133    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880135    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880142    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880144    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880147    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880149    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880152    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880155    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880157    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880160    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880163    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880166    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880169    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880172    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880174    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880177    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880179    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880182    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880185    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:58.885060 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880187    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880189    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880267    2572 flags.go:64] FLAG: --address="0.0.0.0"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880277    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880286    2572 flags.go:64] FLAG: --anonymous-auth="true"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880293    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880300    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880305    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880312    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880316    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880320    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880323    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880326    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880329    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880345    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880352    2572 flags.go:64] FLAG: --cgroup-root=""
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880355    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880358    2572 flags.go:64] FLAG: --client-ca-file=""
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880361    2572 flags.go:64] FLAG: --cloud-provider="external"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880364    2572 flags.go:64] FLAG: --cluster-dns="[]"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880369    2572 flags.go:64] FLAG: --cluster-domain=""
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880372    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880375    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880378    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 20 09:56:58.885507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880381    2572 flags.go:64] FLAG: --container-log-max-files="5"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880385    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880388    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880391    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880394    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880397    2572 flags.go:64] FLAG: --contention-profiling="false"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880401    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880404    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880407    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880410    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880414    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880417    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880420    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880423    2572 flags.go:64] FLAG: --enable-load-reader="false"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880426    2572 flags.go:64] FLAG: --enable-server="true"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880429    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880433    2572 flags.go:64] FLAG: --event-burst="100"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880436    2572 flags.go:64] FLAG: --event-qps="50"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880439    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880442    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880445    2572 flags.go:64] FLAG: --eviction-hard=""
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880449    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880452    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880456    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880459    2572 flags.go:64] FLAG: --eviction-soft=""
Jul 20 09:56:58.886064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880462    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880465    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880468    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880471    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880474    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880477    2572 flags.go:64] FLAG: --fail-swap-on="true"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880480    2572 flags.go:64] FLAG: --feature-gates=""
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880484    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880487    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880490    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880493    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880497    2572 flags.go:64] FLAG: --healthz-port="10248"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880500    2572 flags.go:64] FLAG: --help="false"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880502    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-196.ec2.internal"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880505    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880508    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880511    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880514    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880518    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880520    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880523    2572 flags.go:64] FLAG: --image-service-endpoint=""
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880526    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880529    2572 flags.go:64] FLAG: --kube-api-burst="100"
Jul 20 09:56:58.886659 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880533    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880536    2572 flags.go:64] FLAG: --kube-api-qps="50"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880538    2572 flags.go:64] FLAG: --kube-reserved=""
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880541    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880544    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880547    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880550    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880554    2572 flags.go:64] FLAG: --lock-file=""
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880557    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880559    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880562    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880568    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880570    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880573    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880576    2572 flags.go:64] FLAG: --logging-format="text"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880579    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880582    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880585    2572 flags.go:64] FLAG: --manifest-url=""
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880588    2572 flags.go:64] FLAG: --manifest-url-header=""
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880592    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880595    2572 flags.go:64] FLAG: --max-open-files="1000000"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880599    2572 flags.go:64] FLAG: --max-pods="110"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880602    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880605    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880608    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 20 09:56:58.887262 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880611    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880614    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880617    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880620    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880628    2572 flags.go:64] FLAG: --node-status-max-images="50"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880631    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880634    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880637    2572 flags.go:64] FLAG: --pod-cidr=""
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880640    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880646    2572 flags.go:64] FLAG: --pod-manifest-path=""
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880649    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880652    2572 flags.go:64] FLAG: --pods-per-core="0"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880655    2572 flags.go:64] FLAG: --port="10250"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880658    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880661    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d8d0ed3a4ac0fbd8"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880666    2572 flags.go:64] FLAG: --qos-reserved=""
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880669    2572 flags.go:64] FLAG: --read-only-port="10255"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880672    2572 flags.go:64] FLAG: --register-node="true"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880675    2572 flags.go:64] FLAG: --register-with-taints=""
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880678    2572 flags.go:64] FLAG: --registry-burst="10"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880681    2572 flags.go:64] FLAG: --registry-qps="5"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880684    2572 flags.go:64] FLAG: --reserved-cpus=""
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880687    2572 flags.go:64] FLAG: --reserved-memory=""
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880690    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880693    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 20 09:56:58.887931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880696    2572 flags.go:64] FLAG: --rotate-certificates="false"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880699    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880702    2572 flags.go:64] FLAG: --runonce="false"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880705    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880708    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880710    2572 flags.go:64] FLAG: --seccomp-default="false"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880713    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880716    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880719    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880722    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880725    2572 flags.go:64] FLAG: --storage-driver-password="root"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880728    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880731    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880735    2572 flags.go:64] FLAG: --storage-driver-user="root"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880738    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880741    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880744    2572 flags.go:64] FLAG: --system-cgroups=""
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880747    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880752    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880755    2572 flags.go:64] FLAG: --tls-cert-file=""
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880758    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880763    2572 flags.go:64] FLAG: --tls-min-version=""
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880767    2572 flags.go:64] FLAG: --tls-private-key-file=""
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880770    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880773    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 20 09:56:58.888542 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880775    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880778    2572 flags.go:64] FLAG: --v="2"
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880782    2572 flags.go:64] FLAG: --version="false"
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880789    2572 flags.go:64] FLAG: --vmodule=""
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880793    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.880796    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880916    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880919    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880923    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880925    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880928    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880930    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880933    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880935    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880938    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880940    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880943    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880945    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880948    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880950    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880954    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:58.889146 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880957    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880959    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880962    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880965    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880967    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880970    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880972    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880975    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880979    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880982    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880984    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880987    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880989    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880992    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880994    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.880997    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881000    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881002    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881004    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:58.889663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881007    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881009    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881012    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881014    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881017    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881019    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881022    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881024    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881027    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881029    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881032    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881035    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881039    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881041    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881044    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881047    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881049    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881052    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881055    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881057    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:58.890125 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881060    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881064    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881067    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881070    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881072    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881074    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881077    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881079    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881082    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881085    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881087    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881090    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881092    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881094    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881097    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881100    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881102    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881104    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881107    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:58.890655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881109    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881113    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881117    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881120    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881122    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881126    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881129    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881131    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881134    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881136    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881139    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881142    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881144    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881147    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881150    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881153    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881157    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881160    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881163    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:58.891100 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881166    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881168    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881172    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881187    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881190    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881193    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.881196    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.881201    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.888177    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.888194    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.888215    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.888221    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888287    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888292    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888296    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:58.891655 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888299    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888302    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888305    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888308    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888311    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888314    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888317    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888319    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888322    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888326    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888328    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888331    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888350    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888365    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888367    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888372    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888376    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888379    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:58.892045 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888382    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888385    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888388    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888390    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888393    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888396    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888399    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888402    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888404    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888409    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888411    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888414    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888417    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888422    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888426    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888431    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888434    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888436    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888439    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888441    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:58.892502 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888444    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888446    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888449    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888451    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888458    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888460    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888463    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888467    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888471    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888474    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888476    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888479    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888482    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888484    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888487    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888489    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888491    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888494    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888498    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888502    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:58.892978 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888506    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888511    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888514    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888517    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888519    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888522    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888524    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888527    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888529    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888532    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888534    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888537    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888539    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888542    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888545    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888548    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888550    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888554    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888556    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888559    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:58.893473 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888562    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888564    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888566    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888569    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888572    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888576    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888580    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888584    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888588    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888590    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888593    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888595    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888599    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888604    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888606    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888609    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888612    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:58.893952 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888614    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.888619    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888736    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888742    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888745    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888748    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888751    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888754    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888756    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888760    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888764    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888767    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888769    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888774    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888776    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:58.894370 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888780    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888782    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888785    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888788    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888791    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888793    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888795    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888798    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888800    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888804    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888808    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888812    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888817    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888820    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888822    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888825    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888844    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888848    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888851    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888854    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:58.894747 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888857    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888860    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888863    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888865    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888868    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888872    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888874    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888877    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888880    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888885    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888889    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888895    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888898    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888901    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888904    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888907    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888909    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888912    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888915    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888918    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:58.895218 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888921    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888923    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888926    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888928    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888931    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888934    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888936    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888939    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888941    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888944    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888946    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888948    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888951    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888953    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888956    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888959    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888963    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888968    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888972    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888975    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:58.896090 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888977    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888979    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888982    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888984    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888988    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888991    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888993    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888995    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.888998    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889000    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889003    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889006    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889010    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889013    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889015    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889019    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889023    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889026    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889029    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:57:00.294103 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889032    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889035    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889038    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889042    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889047    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889051    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:56:58.889053    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.889058    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.889160    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.891286    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.892543    2572 server.go:1019] "Starting client certificate rotation"
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.892640    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:56:58.892678    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:57:00.295098 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.294825    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:57:00.296094 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.296078    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:57:00.305632 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.305612    2572 log.go:25] "Validated CRI v1 runtime API"
Jul 20 09:57:00.307656 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.307633    2572 log.go:25] "Validated CRI v1 image API"
Jul 20 09:57:00.308475 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.308459    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 20 09:57:00.313882 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.313862    2572 fs.go:135] Filesystem UUIDs: map[351505bb-3fe5-479c-b4e1-d94677dcd10f:/dev/nvme0n1p3 60fb3a47-9841-4de4-a505-e01c357ac8af:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 20 09:57:00.313944 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.313883    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 20 09:57:00.314651 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.314633    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:57:00.319406 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.319293    2572 manager.go:216] Machine: {Timestamp:2026-07-20 09:57:00.317888723 +0000 UTC m=+0.330990611 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099923 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2037cc636e381a73bb16721a91ed97 SystemUUID:ec2037cc-636e-381a-73bb-16721a91ed97 BootID:ef5ea245-3159-4ff7-8400-d29b59ee3195 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dd:a3:11:d8:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dd:a3:11:d8:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:aa:9f:ed:5a:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 20 09:57:00.319406 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.319397    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 20 09:57:00.319545 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.319477    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 20 09:57:00.322257 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.322227    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 20 09:57:00.322435 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.322261    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-196.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 20 09:57:00.323151 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.323138    2572 topology_manager.go:138] "Creating topology manager with none policy"
Jul 20 09:57:00.323203 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.323157    2572 container_manager_linux.go:309] "Creating device plugin manager"
Jul 20 09:57:00.323203 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.323168    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:57:00.323821 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.323809    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:57:00.325297 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.325286    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 20 09:57:00.328465 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.328451    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:57:00.328598 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.328588    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 20 09:57:00.329135 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.329125    2572 kubelet.go:486] "Attempting to sync node with API server"
Jul 20 09:57:00.329166 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.329139    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 20 09:57:00.329166 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.329158    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 20 09:57:00.329166 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.329166    2572 kubelet.go:398] "Adding apiserver pod source"
Jul 20 09:57:00.329261 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.329176    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 20 09:57:00.329581 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.329567    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:57:00.329627 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.329593    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:57:00.331887 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.331871    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 20 09:57:00.334422 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.334366    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 20 09:57:00.334519 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.334442    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 20 09:57:00.335478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335461    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335482    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335489    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335495    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335504    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335509    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335518    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335523    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335529    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335535    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 20 09:57:00.335539 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335544    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 20 09:57:00.335825 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.335549    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 20 09:57:00.336315 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.336304    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 20 09:57:00.336315 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.336314    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 20 09:57:00.337799 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.337780    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-196.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 20 09:57:00.338309 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.338284    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 20 09:57:00.338357 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.338285    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 20 09:57:00.338707 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.338688    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t6qwc"
Jul 20 09:57:00.339520 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.339507    2572 server.go:1268] "Started kubelet"
Jul 20 09:57:00.339644 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.339607    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 20 09:57:00.339795 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.339605    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 20 09:57:00.339872 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.339812    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 20 09:57:00.340314 ip-10-0-138-196 systemd[1]: Started Kubernetes Kubelet.
Jul 20 09:57:00.340884 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.340866    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 20 09:57:00.341658 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.341642    2572 server.go:310] "Adding debug handlers to kubelet server"
Jul 20 09:57:00.345223 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.344192    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-196.ec2.internal.18c3f6f7864e4301  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-196.ec2.internal,UID:ip-10-0-138-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-196.ec2.internal,},FirstTimestamp:2026-07-20 09:57:00.339479297 +0000 UTC m=+0.352581184,LastTimestamp:2026-07-20 09:57:00.339479297 +0000 UTC m=+0.352581184,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-196.ec2.internal,}"
Jul 20 09:57:00.346383 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.346364    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t6qwc"
Jul 20 09:57:00.346791 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.346773    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 20 09:57:00.347194 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.347176    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 20 09:57:00.347194 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.347189    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 20 09:57:00.347318 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.347265    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:00.347731 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.347713    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 20 09:57:00.347859 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.347794    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 20 09:57:00.347859 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.347808    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 20 09:57:00.347960 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.347928    2572 reconstruct.go:97] "Volume reconstruction finished"
Jul 20 09:57:00.347960 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.347938    2572 reconciler.go:29] "Reconciler: start to sync state"
Jul 20 09:57:00.348050 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.347998    2572 factory.go:55] Registering systemd factory
Jul 20 09:57:00.348050 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.348013    2572 factory.go:223] Registration of the systemd container factory successfully
Jul 20 09:57:00.348273 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.348259    2572 factory.go:153] Registering CRI-O factory
Jul 20 09:57:00.348325 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.348277    2572 factory.go:223] Registration of the crio container factory successfully
Jul 20 09:57:00.348355 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.348328    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 20 09:57:00.348355 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.348347    2572 factory.go:103] Registering Raw factory
Jul 20 09:57:00.348415 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.348359    2572 manager.go:1196] Started watching for new ooms in manager
Jul 20 09:57:00.348823 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.348807    2572 manager.go:349] Starting recovery of all containers
Jul 20 09:57:00.370983 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.370960    2572 manager.go:354] Recovery completed
Jul 20 09:57:00.374795 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.374779    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:00.377667 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.377653    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:00.377892 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.377878    2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-196.ec2.internal\" not found" node="ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.379709 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.379693    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:00.379798 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.379726    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:00.379798 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.379742    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:00.380299 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.380284    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 20 09:57:00.380299 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.380295    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:00.380299 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.380298    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 20 09:57:00.380424 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.380329    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:57:00.381052 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.381038    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:00.381140 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.381069    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:00.381140 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.381080    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:00.381505 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.381493    2572 policy_none.go:49] "None policy: Start"
Jul 20 09:57:00.381541 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.381509    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 20 09:57:00.381541 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.381519    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 20 09:57:00.382603 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.382591    2572 policy_none.go:47] "Start"
Jul 20 09:57:00.417326 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.417186    2572 manager.go:337] "Starting Device Plugin manager"
Jul 20 09:57:00.417420 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.417336    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 20 09:57:00.417420 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.417346    2572 server.go:87] "Starting device plugin registration server"
Jul 20 09:57:00.417581 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.417569    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 20 09:57:00.417626 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.417583    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 20 09:57:00.417701 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.417686    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 20 09:57:00.417778 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.417762    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 20 09:57:00.417778 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.417778    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 20 09:57:00.418279 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.418259    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 20 09:57:00.418360 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.418307    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:00.512471 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.512424    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 20 09:57:00.513816 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.513788    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 20 09:57:00.513931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.513832    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 20 09:57:00.513931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.513867    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 20 09:57:00.513931 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.513908    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 20 09:57:00.517711 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.517695    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:00.517823 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.517695    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:00.518741 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.518726    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:00.518815 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.518758    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:00.518815 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.518776    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:00.518815 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.518798    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.533287 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.533235    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.533287 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.533259    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-196.ec2.internal\": node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:00.574158 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.574139    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:00.614011 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.613977    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal"]
Jul 20 09:57:00.614087 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.614051    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:00.615026 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.615012    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:00.615083 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.615040    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:00.615083 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.615051    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:00.616278 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.616267    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:00.616387 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.616374    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.616423 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.616403    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:00.617359 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.617338    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:00.617461 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.617339    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:00.617461 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.617370    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:00.617461 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.617383    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:00.617461 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.617394    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:00.617461 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.617406    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:00.618522 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.618508    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.618597 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.618530    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:57:00.619267 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.619246    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:57:00.619267 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.619270    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:57:00.619385 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.619282    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:57:00.645176 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.645149    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-196.ec2.internal\" not found" node="ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.649344 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.649325    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0908b916edb3506b0534147bb8444ea6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal\" (UID: \"0908b916edb3506b0534147bb8444ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.649440 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.649351    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0908b916edb3506b0534147bb8444ea6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal\" (UID: \"0908b916edb3506b0534147bb8444ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.649440 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.649368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e1c3e21b15be9fdfd60693e86ae0598-config\") pod \"kube-apiserver-proxy-ip-10-0-138-196.ec2.internal\" (UID: \"9e1c3e21b15be9fdfd60693e86ae0598\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.649513 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.649505    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-196.ec2.internal\" not found" node="ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.675233 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.675213    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:00.749809 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.749779    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0908b916edb3506b0534147bb8444ea6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal\" (UID: \"0908b916edb3506b0534147bb8444ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.749916 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.749811    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0908b916edb3506b0534147bb8444ea6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal\" (UID: \"0908b916edb3506b0534147bb8444ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.749916 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.749835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e1c3e21b15be9fdfd60693e86ae0598-config\") pod \"kube-apiserver-proxy-ip-10-0-138-196.ec2.internal\" (UID: \"9e1c3e21b15be9fdfd60693e86ae0598\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.749916 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.749870    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0908b916edb3506b0534147bb8444ea6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal\" (UID: \"0908b916edb3506b0534147bb8444ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.749916 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.749902    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e1c3e21b15be9fdfd60693e86ae0598-config\") pod \"kube-apiserver-proxy-ip-10-0-138-196.ec2.internal\" (UID: \"9e1c3e21b15be9fdfd60693e86ae0598\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.750067 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.749952    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0908b916edb3506b0534147bb8444ea6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal\" (UID: \"0908b916edb3506b0534147bb8444ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.775912 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.775885    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:00.876691 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.876618    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:00.947871 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.947831    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.953296 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:00.953269    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:00.976709 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:00.976680    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:01.077206 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:01.077172    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:01.177739 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:01.177667    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:01.278243 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:01.278203    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-196.ec2.internal\" not found"
Jul 20 09:57:01.289371 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.289346    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 20 09:57:01.289522 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.289504    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:57:01.289584 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.289519    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:57:01.347113 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.347086    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:01.347881 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.347864    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:01.347969 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.347867    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 20 09:57:01.351736 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.351714    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 09:52:00 +0000 UTC" deadline="2027-12-16 05:26:17.639079205 +0000 UTC"
Jul 20 09:57:01.351776 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.351737    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12331h29m16.287344993s"
Jul 20 09:57:01.362024 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.361992    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:57:01.362192 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.362098    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:01.365879 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.365866    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:57:01.366913 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.366901    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal"
Jul 20 09:57:01.376048 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.376033    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:57:01.389684 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.389661    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-828fv"
Jul 20 09:57:01.397973 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.397954    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-828fv"
Jul 20 09:57:01.555299 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:01.555267    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0908b916edb3506b0534147bb8444ea6.slice/crio-2bf01028aa602bfd126891359793382cf5b79c809e0c14e0db68700e0f762e24 WatchSource:0}: Error finding container 2bf01028aa602bfd126891359793382cf5b79c809e0c14e0db68700e0f762e24: Status 404 returned error can't find the container with id 2bf01028aa602bfd126891359793382cf5b79c809e0c14e0db68700e0f762e24
Jul 20 09:57:01.555663 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:01.555646    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1c3e21b15be9fdfd60693e86ae0598.slice/crio-1010ed1b9a5eaa18784ecc976f349082490a15bc8ec12da36b8dfbc557760c63 WatchSource:0}: Error finding container 1010ed1b9a5eaa18784ecc976f349082490a15bc8ec12da36b8dfbc557760c63: Status 404 returned error can't find the container with id 1010ed1b9a5eaa18784ecc976f349082490a15bc8ec12da36b8dfbc557760c63
Jul 20 09:57:01.560325 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:01.560307    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 09:57:02.330136 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.330095    2572 apiserver.go:52] "Watching apiserver"
Jul 20 09:57:02.340494 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.340461    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 20 09:57:02.341108 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.340913    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-hchgr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd","openshift-dns/node-resolver-2dxpj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal","openshift-multus/network-metrics-daemon-82lt6","openshift-network-diagnostics/network-check-target-9ztdl","openshift-network-operator/iptables-alerter-x8j28","openshift-ovn-kubernetes/ovnkube-node-h4csm","kube-system/global-pull-secret-syncer-n46hj","kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal","openshift-cluster-node-tuning-operator/tuned-g4rfz","openshift-image-registry/node-ca-66rz6","openshift-multus/multus-additional-cni-plugins-6zp58","openshift-multus/multus-lrbz6"]
Jul 20 09:57:02.342669 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.342391    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.344358 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.344332    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.345181 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.345128    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.345288 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.345231    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qnhbr\""
Jul 20 09:57:02.345505 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.345487    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.345842 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.345821    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.348487 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.348110    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:02.348487 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.348263    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2k2zq\""
Jul 20 09:57:02.348487 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.348388    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.348721 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.348514    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 20 09:57:02.348923 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.348900    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.349962 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.349685    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z5h7s\""
Jul 20 09:57:02.349962 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.349725    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.349962 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.349868    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.350187 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.350171    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:02.350272 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.350253    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:02.350806 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.350610    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8n5sn\""
Jul 20 09:57:02.351385 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.351055    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 20 09:57:02.351385 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.351245    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 20 09:57:02.351737 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.351711    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:02.351792 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.351769    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:02.352998 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.352981    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.355638 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.355617    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.355730 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.355693    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.356271 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.356043    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 20 09:57:02.356271 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.356111    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.356271 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.356186    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9vg2s\""
Jul 20 09:57:02.357619 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.356995    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.359642 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.359136    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.360379 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360356    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.360467 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360384    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.360615 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360597    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 20 09:57:02.360795 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360778    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 20 09:57:02.360877 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360595    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b-konnectivity-ca\") pod \"konnectivity-agent-hchgr\" (UID: \"2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b\") " pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:02.360933 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-registration-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.360983 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360932    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-etc-selinux\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.360983 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360954    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-kubernetes\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.360983 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-run\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361150 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.360988    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmrh\" (UniqueName: \"kubernetes.io/projected/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-kube-api-access-kqmrh\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.361150 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysconfig\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361150 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361029    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysctl-conf\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361150 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361046    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-tuned\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361150 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361068    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b-agent-certs\") pod \"konnectivity-agent-hchgr\" (UID: \"2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b\") " pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:02.361150 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361085    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysctl-d\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361150 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361107    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-var-lib-kubelet\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361150 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361147    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bfd919-acee-4caa-ae8b-b48fd31333f4-tmp\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-socket-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361193    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-device-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361214    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbvl\" (UniqueName: \"kubernetes.io/projected/f29da995-4bb0-47b2-9bd5-943406c9c977-kube-api-access-vbbvl\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361265    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-host\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hngf\" (UniqueName: \"kubernetes.io/projected/81bfd919-acee-4caa-ae8b-b48fd31333f4-kube-api-access-7hngf\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361312    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svl8d\" (UniqueName: \"kubernetes.io/projected/852fdd30-79fb-4d25-b6e1-1803dcd85add-kube-api-access-svl8d\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361335    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-hosts-file\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-modprobe-d\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361395    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-lib-modules\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361450    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-sys-fs\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361473    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361496    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-systemd\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.361514 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-sys\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.362096 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-tmp-dir\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.362096 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.361778    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 20 09:57:02.362907 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.362709    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.362907 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.362757    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zbzb8\""
Jul 20 09:57:02.362907 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.362827    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:02.363087 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.362956    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 20 09:57:02.363139 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.363087    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 20 09:57:02.363583 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.363372    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.364581 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.363876    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 20 09:57:02.364581 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.364029    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.364581 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.364297    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.364581 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.364343    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.364915 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.364900    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.365003 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.364977    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwbvg\""
Jul 20 09:57:02.365105 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.365090    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 20 09:57:02.365344 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.365325    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f6vlz\""
Jul 20 09:57:02.365413 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.365391    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 20 09:57:02.366013 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.365802    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 20 09:57:02.366182 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.366166    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hns69\""
Jul 20 09:57:02.400169 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.400018    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:52:01 +0000 UTC" deadline="2028-04-09 21:57:28.900563612 +0000 UTC"
Jul 20 09:57:02.400169 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.400069    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15108h0m26.500498845s"
Jul 20 09:57:02.448860 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.448836    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 20 09:57:02.462448 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462414    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-conf-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.462448 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.462696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.462696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462517    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b-konnectivity-ca\") pod \"konnectivity-agent-hchgr\" (UID: \"2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b\") " pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:02.462696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-etc-selinux\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.462696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462595    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-kubernetes\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.462696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462616    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-run\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.462696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462638    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmrh\" (UniqueName: \"kubernetes.io/projected/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-kube-api-access-kqmrh\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.462696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462687    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-etc-selinux\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.462696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462694    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-kubernetes\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1355235-7340-462d-89ee-3e02019676ee-host-slash\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462746    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-system-cni-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-os-release\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462783    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-run\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysctl-conf\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462820    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-systemd\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-node-log\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462861    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462883    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-kubelet\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-hostroot\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b-agent-certs\") pod \"konnectivity-agent-hchgr\" (UID: \"2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b\") " pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462926    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysctl-conf\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.462959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysctl-d\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463031    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-var-lib-kubelet\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463064 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463068    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysctl-d\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463079    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b-konnectivity-ca\") pod \"konnectivity-agent-hchgr\" (UID: \"2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b\") " pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463068    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf62b\" (UniqueName: \"kubernetes.io/projected/f4785433-f502-4748-a0d9-c718625cdd56-kube-api-access-rf62b\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463107    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-cnibin\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463098    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-var-lib-kubelet\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463122    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-etc-kubernetes\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463139    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blt7m\" (UniqueName: \"kubernetes.io/projected/c443edf8-fe49-4b79-ae03-a78c32a1b5af-kube-api-access-blt7m\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrvp\" (UniqueName: \"kubernetes.io/projected/a1355235-7340-462d-89ee-3e02019676ee-kube-api-access-rsrvp\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463170    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-log-socket\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463184    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-ovnkube-config\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463200    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-cni-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3700312c-9a42-4a43-a2e8-a7827283c09b-kubelet-config\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hngf\" (UniqueName: \"kubernetes.io/projected/81bfd919-acee-4caa-ae8b-b48fd31333f4-kube-api-access-7hngf\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-hosts-file\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463288    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463370    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-hosts-file\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463382    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-slash\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.463710 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-run-netns\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4785433-f502-4748-a0d9-c718625cdd56-ovn-node-metrics-cert\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463464    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0c1a30d-7d22-4667-ac0c-eba67d79c668-cni-binary-copy\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463486    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-cni-multus\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-multus-certs\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463539    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463600    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-sys-fs\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463614    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463622    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-kubelet\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463660    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-ovnkube-script-lib\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463684    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-socket-dir-parent\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463694    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-sys-fs\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463708    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnz7v\" (UniqueName: \"kubernetes.io/projected/a0c1a30d-7d22-4667-ac0c-eba67d79c668-kube-api-access-lnz7v\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-system-cni-dir\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-os-release\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463785    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-sys\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463815    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-systemd-units\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.464285 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463824    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-sys\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1355235-7340-462d-89ee-3e02019676ee-iptables-alerter-script\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463865    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463899    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-registration-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463931    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-etc-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463938    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-registration-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463960    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-ovn\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463980    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-daemon-config\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.463999    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464019    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c443edf8-fe49-4b79-ae03-a78c32a1b5af-serviceca\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464060    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysconfig\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464079    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-tuned\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464115    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-sysconfig\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-cni-netd\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464192    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-cni-bin\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.465035 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-cnibin\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464257    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bfd919-acee-4caa-ae8b-b48fd31333f4-tmp\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-socket-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-device-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464420    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c443edf8-fe49-4b79-ae03-a78c32a1b5af-host\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464447    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbvl\" (UniqueName: \"kubernetes.io/projected/f29da995-4bb0-47b2-9bd5-943406c9c977-kube-api-access-vbbvl\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464463    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-socket-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464474    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-var-lib-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464513    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/852fdd30-79fb-4d25-b6e1-1803dcd85add-device-dir\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464572    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-cni-bin\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464679    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-host\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464774    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svl8d\" (UniqueName: \"kubernetes.io/projected/852fdd30-79fb-4d25-b6e1-1803dcd85add-kube-api-access-svl8d\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464801    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-host\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464804    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464845    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-k8s-cni-cncf-io\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464950    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tf5\" (UniqueName: \"kubernetes.io/projected/72ae4625-f4be-43b5-ab3e-2b10725394da-kube-api-access-x9tf5\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.465765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.464991    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3700312c-9a42-4a43-a2e8-a7827283c09b-dbus\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465019    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-modprobe-d\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465045    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-lib-modules\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465076    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-env-overrides\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465102    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465116    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-modprobe-d\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465127    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-systemd\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465154    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-tmp-dir\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-netns\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.465207    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465223    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-lib-modules\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.465290    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs podName:f29da995-4bb0-47b2-9bd5-943406c9c977 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:02.965255304 +0000 UTC m=+2.978357200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs") pod "network-metrics-daemon-82lt6" (UID: "f29da995-4bb0-47b2-9bd5-943406c9c977") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465388    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-systemd\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.466250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.465576    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-tmp-dir\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.466696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.466634    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bfd919-acee-4caa-ae8b-b48fd31333f4-tmp\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.466696 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.466660    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81bfd919-acee-4caa-ae8b-b48fd31333f4-etc-tuned\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.466931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.466912    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b-agent-certs\") pod \"konnectivity-agent-hchgr\" (UID: \"2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b\") " pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:02.471160 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.471095    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmrh\" (UniqueName: \"kubernetes.io/projected/0991b138-7f5a-47b5-9fa9-436a20cc6b6c-kube-api-access-kqmrh\") pod \"node-resolver-2dxpj\" (UID: \"0991b138-7f5a-47b5-9fa9-436a20cc6b6c\") " pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.471634 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.471615    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hngf\" (UniqueName: \"kubernetes.io/projected/81bfd919-acee-4caa-ae8b-b48fd31333f4-kube-api-access-7hngf\") pod \"tuned-g4rfz\" (UID: \"81bfd919-acee-4caa-ae8b-b48fd31333f4\") " pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.473936 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.473910    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svl8d\" (UniqueName: \"kubernetes.io/projected/852fdd30-79fb-4d25-b6e1-1803dcd85add-kube-api-access-svl8d\") pod \"aws-ebs-csi-driver-node-6l2kd\" (UID: \"852fdd30-79fb-4d25-b6e1-1803dcd85add\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.474100 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.474084    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbvl\" (UniqueName: \"kubernetes.io/projected/f29da995-4bb0-47b2-9bd5-943406c9c977-kube-api-access-vbbvl\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:02.519310 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.519251    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal" event={"ID":"9e1c3e21b15be9fdfd60693e86ae0598","Type":"ContainerStarted","Data":"1010ed1b9a5eaa18784ecc976f349082490a15bc8ec12da36b8dfbc557760c63"}
Jul 20 09:57:02.520635 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.520604    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal" event={"ID":"0908b916edb3506b0534147bb8444ea6","Type":"ContainerStarted","Data":"2bf01028aa602bfd126891359793382cf5b79c809e0c14e0db68700e0f762e24"}
Jul 20 09:57:02.566290 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566249    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566300    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1355235-7340-462d-89ee-3e02019676ee-host-slash\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-system-cni-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566339    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-os-release\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566358    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-systemd\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566379    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-node-log\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566402    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566427    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-kubelet\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566442    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-systemd\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566439    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-system-cni-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.566478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-hostroot\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566488    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566510    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf62b\" (UniqueName: \"kubernetes.io/projected/f4785433-f502-4748-a0d9-c718625cdd56-kube-api-access-rf62b\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566526    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-kubelet\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566493    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-hostroot\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566540    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-cnibin\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566548    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-node-log\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566400    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1355235-7340-462d-89ee-3e02019676ee-host-slash\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-etc-kubernetes\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566609    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blt7m\" (UniqueName: \"kubernetes.io/projected/c443edf8-fe49-4b79-ae03-a78c32a1b5af-kube-api-access-blt7m\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566617    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-os-release\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566622    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-cnibin\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566633    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrvp\" (UniqueName: \"kubernetes.io/projected/a1355235-7340-462d-89ee-3e02019676ee-kube-api-access-rsrvp\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566655    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-etc-kubernetes\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-log-socket\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566689    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-log-socket\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566695    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-ovnkube-config\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566720    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-cni-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567030 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3700312c-9a42-4a43-a2e8-a7827283c09b-kubelet-config\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-slash\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-run-netns\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4785433-f502-4748-a0d9-c718625cdd56-ovn-node-metrics-cert\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566848    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0c1a30d-7d22-4667-ac0c-eba67d79c668-cni-binary-copy\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566848    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3700312c-9a42-4a43-a2e8-a7827283c09b-kubelet-config\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566872    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-cni-multus\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566894    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-multus-certs\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566899    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-run-netns\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566921    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-kubelet\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566926    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-slash\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566950    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566961    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-ovnkube-script-lib\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566979    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-cni-multus\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566983    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-cni-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.566991    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-socket-dir-parent\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnz7v\" (UniqueName: \"kubernetes.io/projected/a0c1a30d-7d22-4667-ac0c-eba67d79c668-kube-api-access-lnz7v\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.567935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567019    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-kubelet\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-system-cni-dir\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567062    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-os-release\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567074    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-socket-dir-parent\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567086    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-systemd-units\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1355235-7340-462d-89ee-3e02019676ee-iptables-alerter-script\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567119    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-system-cni-dir\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567139    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-etc-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-ovn\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-daemon-config\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567232    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-ovnkube-config\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567297    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-os-release\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567334    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567343    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-ovn\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.567382    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:02.568646 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567408    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-systemd-units\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.567449    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret podName:3700312c-9a42-4a43-a2e8-a7827283c09b nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.067431007 +0000 UTC m=+3.080532889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret") pod "global-pull-secret-syncer-n46hj" (UID: "3700312c-9a42-4a43-a2e8-a7827283c09b") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567467    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0c1a30d-7d22-4667-ac0c-eba67d79c668-cni-binary-copy\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567489    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-ovnkube-script-lib\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567540    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-etc-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567024    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-multus-certs\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567598    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c443edf8-fe49-4b79-ae03-a78c32a1b5af-serviceca\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567649    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-cni-netd\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567676    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-cni-bin\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567700    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-cnibin\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567711    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-cni-netd\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567717    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567726    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c443edf8-fe49-4b79-ae03-a78c32a1b5af-host\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567769    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-cnibin\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-var-lib-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567787    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c443edf8-fe49-4b79-ae03-a78c32a1b5af-host\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567804    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-var-lib-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569216 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567806    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-cni-bin\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567824    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-daemon-config\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567838    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-var-lib-cni-bin\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567844    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-cni-bin\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567877    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567936    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-k8s-cni-cncf-io\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567961    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tf5\" (UniqueName: \"kubernetes.io/projected/72ae4625-f4be-43b5-ab3e-2b10725394da-kube-api-access-x9tf5\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.567985    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3700312c-9a42-4a43-a2e8-a7827283c09b-dbus\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-env-overrides\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568033    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568054    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568073    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c443edf8-fe49-4b79-ae03-a78c32a1b5af-serviceca\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568008    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1355235-7340-462d-89ee-3e02019676ee-iptables-alerter-script\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568121    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3700312c-9a42-4a43-a2e8-a7827283c09b-dbus\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568138    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-k8s-cni-cncf-io\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568148    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-netns\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.569844 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568179    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-conf-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.570339 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568203    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.570339 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568249    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-host-run-netns\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.570339 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568274    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0c1a30d-7d22-4667-ac0c-eba67d79c668-multus-conf-dir\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.570339 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568326    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72ae4625-f4be-43b5-ab3e-2b10725394da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.570339 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568300    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4785433-f502-4748-a0d9-c718625cdd56-run-openvswitch\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.570339 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568388    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72ae4625-f4be-43b5-ab3e-2b10725394da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.570339 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.568451    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4785433-f502-4748-a0d9-c718625cdd56-env-overrides\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.570339 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.569652    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4785433-f502-4748-a0d9-c718625cdd56-ovn-node-metrics-cert\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.582080 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.582020    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blt7m\" (UniqueName: \"kubernetes.io/projected/c443edf8-fe49-4b79-ae03-a78c32a1b5af-kube-api-access-blt7m\") pod \"node-ca-66rz6\" (UID: \"c443edf8-fe49-4b79-ae03-a78c32a1b5af\") " pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.586086 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.586058    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:02.586228 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.586092    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:02.586228 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.586125    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pclxh for pod openshift-network-diagnostics/network-check-target-9ztdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:02.586718 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.586627    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrvp\" (UniqueName: \"kubernetes.io/projected/a1355235-7340-462d-89ee-3e02019676ee-kube-api-access-rsrvp\") pod \"iptables-alerter-x8j28\" (UID: \"a1355235-7340-462d-89ee-3e02019676ee\") " pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.586889 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.586868    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh podName:1e0149b6-9f1d-43fd-bca3-174fbee60711 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.086249906 +0000 UTC m=+3.099351780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pclxh" (UniqueName: "kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh") pod "network-check-target-9ztdl" (UID: "1e0149b6-9f1d-43fd-bca3-174fbee60711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:02.586889 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.586865    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf62b\" (UniqueName: \"kubernetes.io/projected/f4785433-f502-4748-a0d9-c718625cdd56-kube-api-access-rf62b\") pod \"ovnkube-node-h4csm\" (UID: \"f4785433-f502-4748-a0d9-c718625cdd56\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.587047 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.586867    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnz7v\" (UniqueName: \"kubernetes.io/projected/a0c1a30d-7d22-4667-ac0c-eba67d79c668-kube-api-access-lnz7v\") pod \"multus-lrbz6\" (UID: \"a0c1a30d-7d22-4667-ac0c-eba67d79c668\") " pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.588256 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.588236    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tf5\" (UniqueName: \"kubernetes.io/projected/72ae4625-f4be-43b5-ab3e-2b10725394da-kube-api-access-x9tf5\") pod \"multus-additional-cni-plugins-6zp58\" (UID: \"72ae4625-f4be-43b5-ab3e-2b10725394da\") " pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.654232 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.654196    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g4rfz"
Jul 20 09:57:02.665260 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.665238    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd"
Jul 20 09:57:02.674775 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.674750    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2dxpj"
Jul 20 09:57:02.681443 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.681417    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:02.689971 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.689947    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x8j28"
Jul 20 09:57:02.697708 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.697682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:02.706108 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.706088    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lrbz6"
Jul 20 09:57:02.715574 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.715541    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zp58"
Jul 20 09:57:02.721094 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.721074    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-66rz6"
Jul 20 09:57:02.754767 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.754737    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:02.788913 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.788886    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:02.970483 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:02.970399    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:02.970680 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.970506    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.970680 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:02.970588    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs podName:f29da995-4bb0-47b2-9bd5-943406c9c977 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.970568038 +0000 UTC m=+3.983669926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs") pod "network-metrics-daemon-82lt6" (UID: "f29da995-4bb0-47b2-9bd5-943406c9c977") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:03.071779 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.071713    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:03.071961 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.071911    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:03.072024 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.071992    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret podName:3700312c-9a42-4a43-a2e8-a7827283c09b nodeName:}" failed. No retries permitted until 2026-07-20 09:57:04.071971309 +0000 UTC m=+4.085073192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret") pod "global-pull-secret-syncer-n46hj" (UID: "3700312c-9a42-4a43-a2e8-a7827283c09b") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:03.172230 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.172192    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:03.172412 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.172390    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:03.172483 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.172421    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:03.172483 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.172436    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pclxh for pod openshift-network-diagnostics/network-check-target-9ztdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:03.172584 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.172494    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh podName:1e0149b6-9f1d-43fd-bca3-174fbee60711 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:04.172475708 +0000 UTC m=+4.185577603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclxh" (UniqueName: "kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh") pod "network-check-target-9ztdl" (UID: "1e0149b6-9f1d-43fd-bca3-174fbee60711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:03.400691 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.400578    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:52:01 +0000 UTC" deadline="2028-03-14 17:48:50.082268261 +0000 UTC"
Jul 20 09:57:03.400691 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.400633    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14479h51m46.681640222s"
Jul 20 09:57:03.495538 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:03.494906    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0991b138_7f5a_47b5_9fa9_436a20cc6b6c.slice/crio-377784d75ada2498443a3b46f79f525e4ee21e5924628a93b9dd91149a12476d WatchSource:0}: Error finding container 377784d75ada2498443a3b46f79f525e4ee21e5924628a93b9dd91149a12476d: Status 404 returned error can't find the container with id 377784d75ada2498443a3b46f79f525e4ee21e5924628a93b9dd91149a12476d
Jul 20 09:57:03.496245 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:03.495977    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc443edf8_fe49_4b79_ae03_a78c32a1b5af.slice/crio-40c632f4fe406227fbda6d82423c645a0397ddfcac859af81aad5ec071f6696c WatchSource:0}: Error finding container 40c632f4fe406227fbda6d82423c645a0397ddfcac859af81aad5ec071f6696c: Status 404 returned error can't find the container with id 40c632f4fe406227fbda6d82423c645a0397ddfcac859af81aad5ec071f6696c
Jul 20 09:57:03.497708 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:03.497634    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bfd919_acee_4caa_ae8b_b48fd31333f4.slice/crio-9b87284ee3899f39636d3d9cc87d075151c583094c42b24070c1db0452a24be6 WatchSource:0}: Error finding container 9b87284ee3899f39636d3d9cc87d075151c583094c42b24070c1db0452a24be6: Status 404 returned error can't find the container with id 9b87284ee3899f39636d3d9cc87d075151c583094c42b24070c1db0452a24be6
Jul 20 09:57:03.499914 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:03.499806    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4785433_f502_4748_a0d9_c718625cdd56.slice/crio-5c4d6d73b5f201e4129cf39b5ca62ddaea0a3b863d76df036dbebbfde08448c8 WatchSource:0}: Error finding container 5c4d6d73b5f201e4129cf39b5ca62ddaea0a3b863d76df036dbebbfde08448c8: Status 404 returned error can't find the container with id 5c4d6d73b5f201e4129cf39b5ca62ddaea0a3b863d76df036dbebbfde08448c8
Jul 20 09:57:03.502744 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:03.501544    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c1a30d_7d22_4667_ac0c_eba67d79c668.slice/crio-50d27deb22b07dbe19e13521ef447acb4495a80f9d8f5487995bd4aec4e4025b WatchSource:0}: Error finding container 50d27deb22b07dbe19e13521ef447acb4495a80f9d8f5487995bd4aec4e4025b: Status 404 returned error can't find the container with id 50d27deb22b07dbe19e13521ef447acb4495a80f9d8f5487995bd4aec4e4025b
Jul 20 09:57:03.504250 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:03.503384    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852fdd30_79fb_4d25_b6e1_1803dcd85add.slice/crio-edf1171f7c680b4017900841381ff414573ad2e50da40d38d3a09917b99993fe WatchSource:0}: Error finding container edf1171f7c680b4017900841381ff414573ad2e50da40d38d3a09917b99993fe: Status 404 returned error can't find the container with id edf1171f7c680b4017900841381ff414573ad2e50da40d38d3a09917b99993fe
Jul 20 09:57:03.505360 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:03.505098    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1355235_7340_462d_89ee_3e02019676ee.slice/crio-b5e9a1576e9e5b466e11442988d2a098a128261d1e1962675faf9c50ff9d2b68 WatchSource:0}: Error finding container b5e9a1576e9e5b466e11442988d2a098a128261d1e1962675faf9c50ff9d2b68: Status 404 returned error can't find the container with id b5e9a1576e9e5b466e11442988d2a098a128261d1e1962675faf9c50ff9d2b68
Jul 20 09:57:03.505661 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:03.505641    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ae4625_f4be_43b5_ab3e_2b10725394da.slice/crio-b81ccaa174c60fa4feeb67e896d1309130be9541ff985e5eaec1f57bd4cddba7 WatchSource:0}: Error finding container b81ccaa174c60fa4feeb67e896d1309130be9541ff985e5eaec1f57bd4cddba7: Status 404 returned error can't find the container with id b81ccaa174c60fa4feeb67e896d1309130be9541ff985e5eaec1f57bd4cddba7
Jul 20 09:57:03.514379 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.514358    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:03.514493 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.514471    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:03.523380 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.523354    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerStarted","Data":"b81ccaa174c60fa4feeb67e896d1309130be9541ff985e5eaec1f57bd4cddba7"}
Jul 20 09:57:03.524537 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.524514    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"5c4d6d73b5f201e4129cf39b5ca62ddaea0a3b863d76df036dbebbfde08448c8"}
Jul 20 09:57:03.525954 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.525931    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g4rfz" event={"ID":"81bfd919-acee-4caa-ae8b-b48fd31333f4","Type":"ContainerStarted","Data":"9b87284ee3899f39636d3d9cc87d075151c583094c42b24070c1db0452a24be6"}
Jul 20 09:57:03.526914 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.526892    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-66rz6" event={"ID":"c443edf8-fe49-4b79-ae03-a78c32a1b5af","Type":"ContainerStarted","Data":"40c632f4fe406227fbda6d82423c645a0397ddfcac859af81aad5ec071f6696c"}
Jul 20 09:57:03.527811 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.527789    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2dxpj" event={"ID":"0991b138-7f5a-47b5-9fa9-436a20cc6b6c","Type":"ContainerStarted","Data":"377784d75ada2498443a3b46f79f525e4ee21e5924628a93b9dd91149a12476d"}
Jul 20 09:57:03.528835 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.528816    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lrbz6" event={"ID":"a0c1a30d-7d22-4667-ac0c-eba67d79c668","Type":"ContainerStarted","Data":"50d27deb22b07dbe19e13521ef447acb4495a80f9d8f5487995bd4aec4e4025b"}
Jul 20 09:57:03.529828 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.529810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hchgr" event={"ID":"2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b","Type":"ContainerStarted","Data":"1bab5f1a0a1ff85bbcd60431733f05f08421b5756416803f072661b3b8008592"}
Jul 20 09:57:03.530804 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.530787    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x8j28" event={"ID":"a1355235-7340-462d-89ee-3e02019676ee","Type":"ContainerStarted","Data":"b5e9a1576e9e5b466e11442988d2a098a128261d1e1962675faf9c50ff9d2b68"}
Jul 20 09:57:03.531972 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.531919    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd" event={"ID":"852fdd30-79fb-4d25-b6e1-1803dcd85add","Type":"ContainerStarted","Data":"edf1171f7c680b4017900841381ff414573ad2e50da40d38d3a09917b99993fe"}
Jul 20 09:57:03.979167 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:03.978768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:03.979167 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.979088    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:03.979167 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:03.979168    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs podName:f29da995-4bb0-47b2-9bd5-943406c9c977 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:05.979148264 +0000 UTC m=+5.992250151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs") pod "network-metrics-daemon-82lt6" (UID: "f29da995-4bb0-47b2-9bd5-943406c9c977") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:04.080095 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:04.079904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:04.080273 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:04.080101    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:04.080273 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:04.080175    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret podName:3700312c-9a42-4a43-a2e8-a7827283c09b nodeName:}" failed. No retries permitted until 2026-07-20 09:57:06.080156621 +0000 UTC m=+6.093258497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret") pod "global-pull-secret-syncer-n46hj" (UID: "3700312c-9a42-4a43-a2e8-a7827283c09b") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:04.180332 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:04.180292    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:04.180528 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:04.180513    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:04.180613 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:04.180536    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:04.180613 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:04.180548    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pclxh for pod openshift-network-diagnostics/network-check-target-9ztdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:04.180733 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:04.180622    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh podName:1e0149b6-9f1d-43fd-bca3-174fbee60711 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:06.180602642 +0000 UTC m=+6.193704519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclxh" (UniqueName: "kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh") pod "network-check-target-9ztdl" (UID: "1e0149b6-9f1d-43fd-bca3-174fbee60711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:04.517536 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:04.517503    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:04.518127 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:04.517652    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:04.518127 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:04.518040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:04.518241 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:04.518138    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:04.574125 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:04.574095    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal" event={"ID":"9e1c3e21b15be9fdfd60693e86ae0598","Type":"ContainerStarted","Data":"98fc028d520f8800c7d8c6f76c473675e000d03da593c181394f721375ba477a"}
Jul 20 09:57:04.592491 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:04.592434    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-196.ec2.internal" podStartSLOduration=3.592415253 podStartE2EDuration="3.592415253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:04.591850822 +0000 UTC m=+4.604952719" watchObservedRunningTime="2026-07-20 09:57:04.592415253 +0000 UTC m=+4.605517151"
Jul 20 09:57:04.594432 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:04.594400    2572 generic.go:356] "Generic (PLEG): container finished" podID="0908b916edb3506b0534147bb8444ea6" containerID="61df271e9faecbb509d772665c8770dcc816e3490633c8e5fb6747bed6fa2cdf" exitCode=0
Jul 20 09:57:04.594585 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:04.594451    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal" event={"ID":"0908b916edb3506b0534147bb8444ea6","Type":"ContainerDied","Data":"61df271e9faecbb509d772665c8770dcc816e3490633c8e5fb6747bed6fa2cdf"}
Jul 20 09:57:05.514119 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:05.514080    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:05.514290 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:05.514227    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:05.616692 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:05.616608    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal" event={"ID":"0908b916edb3506b0534147bb8444ea6","Type":"ContainerStarted","Data":"99d14dbc920e29a218160e3b4468c5cb2ed374488854c606d21a7a5a77a7588b"}
Jul 20 09:57:05.995299 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:05.995261    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:05.995471 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:05.995409    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:05.995546 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:05.995474    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs podName:f29da995-4bb0-47b2-9bd5-943406c9c977 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:09.995454406 +0000 UTC m=+10.008556290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs") pod "network-metrics-daemon-82lt6" (UID: "f29da995-4bb0-47b2-9bd5-943406c9c977") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:06.095831 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:06.095792    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:06.096002 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:06.095944    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:06.096074 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:06.096005    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret podName:3700312c-9a42-4a43-a2e8-a7827283c09b nodeName:}" failed. No retries permitted until 2026-07-20 09:57:10.095987395 +0000 UTC m=+10.109089272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret") pod "global-pull-secret-syncer-n46hj" (UID: "3700312c-9a42-4a43-a2e8-a7827283c09b") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:06.196729 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:06.196691    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:06.196894 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:06.196845    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:06.196894 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:06.196868    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:06.196894 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:06.196880    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pclxh for pod openshift-network-diagnostics/network-check-target-9ztdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:06.197046 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:06.196940    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh podName:1e0149b6-9f1d-43fd-bca3-174fbee60711 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:10.19691868 +0000 UTC m=+10.210020554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclxh" (UniqueName: "kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh") pod "network-check-target-9ztdl" (UID: "1e0149b6-9f1d-43fd-bca3-174fbee60711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:06.514736 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:06.514107    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:06.514736 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:06.514241    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:06.514736 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:06.514301    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:06.514736 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:06.514386    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:07.514995 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:07.514464    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:07.514995 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:07.514656    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:08.514492 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:08.514439    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:08.514693 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:08.514583    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:08.514939 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:08.514439    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:08.514939 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:08.514901    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:09.514798 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:09.514308    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:09.514798 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:09.514437    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:10.034946 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:10.034907    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:10.035426 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.035048    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:10.035426 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.035114    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs podName:f29da995-4bb0-47b2-9bd5-943406c9c977 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:18.035094124 +0000 UTC m=+18.048196002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs") pod "network-metrics-daemon-82lt6" (UID: "f29da995-4bb0-47b2-9bd5-943406c9c977") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:10.135841 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:10.135803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:10.136020 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.135972    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:10.136069 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.136049    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret podName:3700312c-9a42-4a43-a2e8-a7827283c09b nodeName:}" failed. No retries permitted until 2026-07-20 09:57:18.136027419 +0000 UTC m=+18.149129315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret") pod "global-pull-secret-syncer-n46hj" (UID: "3700312c-9a42-4a43-a2e8-a7827283c09b") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:10.236743 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:10.236704    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:10.236931 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.236907    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:10.237008 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.236937    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:10.237008 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.236951    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pclxh for pod openshift-network-diagnostics/network-check-target-9ztdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:10.237121 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.237023    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh podName:1e0149b6-9f1d-43fd-bca3-174fbee60711 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:18.236994756 +0000 UTC m=+18.250096651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclxh" (UniqueName: "kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh") pod "network-check-target-9ztdl" (UID: "1e0149b6-9f1d-43fd-bca3-174fbee60711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:10.515378 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:10.515331    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:10.515378 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:10.515368    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:10.515622 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.515445    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:10.515817 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:10.515789    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:11.514798 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:11.514757    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:11.515243 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:11.514895    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:12.514604 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:12.514566    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:12.514788 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:12.514710    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:12.514999 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:12.514977    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:12.515343 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:12.515114    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:13.514386 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:13.514348    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:13.514532 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:13.514493    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:14.514906 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:14.514870    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:14.515351 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:14.514928    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:14.515351 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:14.515059    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:14.515351 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:14.515167    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:15.514747 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:15.514705    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:15.514944 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:15.514847    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:16.515091 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:16.515056    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:16.515534 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:16.515099    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:16.515534 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:16.515172    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:16.515534 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:16.515305    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:17.514166 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:17.514134    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:17.514396 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:17.514268    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:18.097291 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:18.097245    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:18.097812 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.097385    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:18.097812 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.097481    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs podName:f29da995-4bb0-47b2-9bd5-943406c9c977 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.097460146 +0000 UTC m=+34.110562021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs") pod "network-metrics-daemon-82lt6" (UID: "f29da995-4bb0-47b2-9bd5-943406c9c977") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:18.197605 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:18.197568    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:18.197766 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.197701    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:18.197766 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.197765    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret podName:3700312c-9a42-4a43-a2e8-a7827283c09b nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.19774825 +0000 UTC m=+34.210850130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret") pod "global-pull-secret-syncer-n46hj" (UID: "3700312c-9a42-4a43-a2e8-a7827283c09b") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:18.298283 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:18.298249    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:18.298453 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.298376    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:18.298453 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.298394    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:18.298453 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.298404    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pclxh for pod openshift-network-diagnostics/network-check-target-9ztdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:18.298606 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.298455    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh podName:1e0149b6-9f1d-43fd-bca3-174fbee60711 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.298436783 +0000 UTC m=+34.311538658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclxh" (UniqueName: "kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh") pod "network-check-target-9ztdl" (UID: "1e0149b6-9f1d-43fd-bca3-174fbee60711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:18.515021 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:18.514987    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:18.515196 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:18.514987    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:18.515196 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.515121    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:18.515310 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:18.515255    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:19.514693 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:19.514655    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:19.515159 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:19.514794    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:20.515517 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:20.515476    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:20.515945 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:20.515522    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:20.515945 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:20.515616    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:20.515945 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:20.515745    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:21.514500 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:21.514468    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:21.514684 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:21.514599    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:22.517447 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.517419    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:22.517830 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:22.517544    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:22.517830 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.517735    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:22.518001 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:22.517834    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:22.651500 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.651311    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g4rfz" event={"ID":"81bfd919-acee-4caa-ae8b-b48fd31333f4","Type":"ContainerStarted","Data":"6779c9b72c2c9950c0371f3d71dc43eaadd47666464bf73de072d185e4c6ad81"}
Jul 20 09:57:22.653159 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.653129    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-66rz6" event={"ID":"c443edf8-fe49-4b79-ae03-a78c32a1b5af","Type":"ContainerStarted","Data":"89bfec9265e86ed6981a556e7e50a8c9643f40a129774856e735965556509c6a"}
Jul 20 09:57:22.654507 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.654484    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2dxpj" event={"ID":"0991b138-7f5a-47b5-9fa9-436a20cc6b6c","Type":"ContainerStarted","Data":"ebabbe257837ff3757c537d13a02d1634bc4a31eb0269a9ba49aa3fcb5bd1b17"}
Jul 20 09:57:22.655853 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.655835    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lrbz6" event={"ID":"a0c1a30d-7d22-4667-ac0c-eba67d79c668","Type":"ContainerStarted","Data":"ebb0ec7ed057751fa0983322e0d0cddc74158d648bbfb36aa4cddcbdbc6dc8e7"}
Jul 20 09:57:22.657252 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.657232    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hchgr" event={"ID":"2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b","Type":"ContainerStarted","Data":"bdcc00a1208b0b72243f11d9322ca626f70067df0d6daef4a69dafd86e5c2cf1"}
Jul 20 09:57:22.658584 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.658550    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd" event={"ID":"852fdd30-79fb-4d25-b6e1-1803dcd85add","Type":"ContainerStarted","Data":"d90a136cc0faec24d8b62d798b1e4256d82d7f95f649c21b4cf0ed3000ac679e"}
Jul 20 09:57:22.659901 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.659883    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerStarted","Data":"c7b0044a77ea3b3c0ac858def4e25e8b763bf93dd99299a1bea3f6315493deaf"}
Jul 20 09:57:22.686009 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.685965    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g4rfz" podStartSLOduration=3.79808786 podStartE2EDuration="22.685946801s" totalImagesPullingTime="18.887858941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.68565693 +0000 UTC m=+22.698758826" watchObservedRunningTime="2026-07-20 09:57:22.685946801 +0000 UTC m=+22.699048698"
Jul 20 09:57:22.686404 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.686376    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-196.ec2.internal" podStartSLOduration=21.68636898 podStartE2EDuration="21.68636898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:05.633545163 +0000 UTC m=+5.646647059" watchObservedRunningTime="2026-07-20 09:57:22.68636898 +0000 UTC m=+22.699470878"
Jul 20 09:57:22.701772 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.701733    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hchgr" podStartSLOduration=12.07424401 podStartE2EDuration="22.701717318s" totalImagesPullingTime="10.627473308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.701465404 +0000 UTC m=+22.714567307" watchObservedRunningTime="2026-07-20 09:57:22.701717318 +0000 UTC m=+22.714819214"
Jul 20 09:57:22.745638 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.745523    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-66rz6" podStartSLOduration=2.876627638 podStartE2EDuration="21.745502832s" totalImagesPullingTime="18.868875194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.74534392 +0000 UTC m=+22.758445816" watchObservedRunningTime="2026-07-20 09:57:22.745502832 +0000 UTC m=+22.758604728"
Jul 20 09:57:22.760181 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.760137    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2dxpj" podStartSLOduration=3.872072364 podStartE2EDuration="22.760122596s" totalImagesPullingTime="18.888050232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.759829298 +0000 UTC m=+22.772931194" watchObservedRunningTime="2026-07-20 09:57:22.760122596 +0000 UTC m=+22.773224493"
Jul 20 09:57:22.966447 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.966420    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:22.966941 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.966922    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:22.983407 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:22.983373    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lrbz6" podStartSLOduration=4.087334145 podStartE2EDuration="22.983358358s" totalImagesPullingTime="18.896024213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.781143237 +0000 UTC m=+22.794245134" watchObservedRunningTime="2026-07-20 09:57:22.983358358 +0000 UTC m=+22.996460253"
Jul 20 09:57:23.515157 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.514992    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:23.515250 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:23.515225    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:23.663472 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.663441    2572 generic.go:356] "Generic (PLEG): container finished" podID="72ae4625-f4be-43b5-ab3e-2b10725394da" containerID="c7b0044a77ea3b3c0ac858def4e25e8b763bf93dd99299a1bea3f6315493deaf" exitCode=0
Jul 20 09:57:23.664296 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.663548    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerDied","Data":"c7b0044a77ea3b3c0ac858def4e25e8b763bf93dd99299a1bea3f6315493deaf"}
Jul 20 09:57:23.666790 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.666772    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 09:57:23.667116 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.667089    2572 generic.go:356] "Generic (PLEG): container finished" podID="f4785433-f502-4748-a0d9-c718625cdd56" containerID="a19f8140340e8d069f2f7ab53fc1065d90d9804f3571580b709834cf5d7a1db4" exitCode=1
Jul 20 09:57:23.667615 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.667593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"69645b543557cafc0c1507326c3943606a8956c65a326b7a49ff15b8deb62c49"}
Jul 20 09:57:23.667697 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.667670    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"c62316a38254ca1b3fcbc3c3d7c8a6ce468ffb4df3bc70dbb918a518d155d677"}
Jul 20 09:57:23.667748 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.667718    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"ddc8bd313f3701393d02c6e26ed189a0d5f0aa92408a708e5b7946095a723762"}
Jul 20 09:57:23.667799 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.667769    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"18388caaf6b1ab40de6be62de9321403de4058f11f3aafdc0bf5789fb1e7e9d7"}
Jul 20 09:57:23.667892 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.667816    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerDied","Data":"a19f8140340e8d069f2f7ab53fc1065d90d9804f3571580b709834cf5d7a1db4"}
Jul 20 09:57:23.667892 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.667873    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"c8b4811153751c161958bb2e75307144956a8dae3e03b96f1c76602e8319807f"}
Jul 20 09:57:23.668905 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.668679    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:23.669581 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.669366    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hchgr"
Jul 20 09:57:23.738249 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:23.738222    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 20 09:57:24.427948 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:24.427769    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T09:57:23.738238856Z","UUID":"03b6130f-ac65-4f07-b5b9-a557f4b72286","Handler":null,"Name":"","Endpoint":""}
Jul 20 09:57:24.429420 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:24.429401    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 20 09:57:24.429550 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:24.429442    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 20 09:57:24.514632 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:24.514597    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:24.514819 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:24.514604    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:24.514819 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:24.514723    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:24.514937 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:24.514835    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:24.672295 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:24.672260    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x8j28" event={"ID":"a1355235-7340-462d-89ee-3e02019676ee","Type":"ContainerStarted","Data":"e398d1c604ebcc1a364d03c39e027ab45c79a690cd756fe8cd4e5f79e750af95"}
Jul 20 09:57:24.674638 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:24.674577    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd" event={"ID":"852fdd30-79fb-4d25-b6e1-1803dcd85add","Type":"ContainerStarted","Data":"6429ce7a9771ce85c3d87a638cdcff899432db30424f7bec59f604cde9e9058f"}
Jul 20 09:57:24.687976 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:24.687861    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x8j28" podStartSLOduration=5.827662339 podStartE2EDuration="24.687844712s" totalImagesPullingTime="18.860182373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:24.687824635 +0000 UTC m=+24.700926544" watchObservedRunningTime="2026-07-20 09:57:24.687844712 +0000 UTC m=+24.700946608"
Jul 20 09:57:25.514585 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:25.514483    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:25.514757 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:25.514620    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:25.681532 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:25.681493    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 09:57:25.681963 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:25.681930    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"9363f752c60d5b4f6ae2e401f3037e451bf351e8e72b1a4a6dc9193737e6a1d1"}
Jul 20 09:57:25.684204 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:25.684142    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd" event={"ID":"852fdd30-79fb-4d25-b6e1-1803dcd85add","Type":"ContainerStarted","Data":"13e76e8ca0805e20f8b45032e93e16b7c7786eceb629606879c96649cd39a9d4"}
Jul 20 09:57:25.703789 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:25.703735    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6l2kd" podStartSLOduration=4.299527604 podStartE2EDuration="25.703713418s" totalImagesPullingTime="21.404185814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:25.703670148 +0000 UTC m=+25.716772043" watchObservedRunningTime="2026-07-20 09:57:25.703713418 +0000 UTC m=+25.716815311"
Jul 20 09:57:26.514632 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:26.514367    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:26.514810 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:26.514439    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:26.514810 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:26.514741    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:26.514915 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:26.514835    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:27.514796 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:27.514760    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:27.515323 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:27.514883    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:28.514824 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.514646    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:28.515448 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.514697    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:28.515448 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:28.514903    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:28.515448 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:28.514971    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:28.693685 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.693606    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 09:57:28.694039 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.694007    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"e3f83d45ca8877f3b66e3a9be469a0ebf69babceb095b30d4b0cb8531714e7ad"}
Jul 20 09:57:28.694232 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.694209    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:28.694414 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.694391    2572 scope.go:117] "RemoveContainer" containerID="a19f8140340e8d069f2f7ab53fc1065d90d9804f3571580b709834cf5d7a1db4"
Jul 20 09:57:28.695819 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.695800    2572 generic.go:356] "Generic (PLEG): container finished" podID="72ae4625-f4be-43b5-ab3e-2b10725394da" containerID="1af80892044daa5f2bf6964282fc8d2cb6fa99dfeaaaf4b0cee5db30983c775d" exitCode=0
Jul 20 09:57:28.695891 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.695830    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerDied","Data":"1af80892044daa5f2bf6964282fc8d2cb6fa99dfeaaaf4b0cee5db30983c775d"}
Jul 20 09:57:28.711059 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:28.711031    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:29.514416 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.514350    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:29.514539 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:29.514509    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:29.699774 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.699609    2572 generic.go:356] "Generic (PLEG): container finished" podID="72ae4625-f4be-43b5-ab3e-2b10725394da" containerID="ccf47c05fe1d0393b0a0e1ddb1ee4d9f5d0fd48beec775ee6865927c698cac94" exitCode=0
Jul 20 09:57:29.700140 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.699687    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerDied","Data":"ccf47c05fe1d0393b0a0e1ddb1ee4d9f5d0fd48beec775ee6865927c698cac94"}
Jul 20 09:57:29.703589 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.703538    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 09:57:29.703934 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.703916    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" event={"ID":"f4785433-f502-4748-a0d9-c718625cdd56","Type":"ContainerStarted","Data":"862a15697ffd18dcaf8f6556a581075ad147c17a55df3cd2ae639cb2c06b8382"}
Jul 20 09:57:29.704032 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.704001    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 20 09:57:29.704250 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.704236    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:29.718363 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.718343    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:29.787735 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.787657    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n46hj"]
Jul 20 09:57:29.787841 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.787811    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:29.787930 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:29.787910    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:29.788202 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.788183    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9ztdl"]
Jul 20 09:57:29.788282 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.788272    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:29.788390 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:29.788359    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:29.789661 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.789638    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-82lt6"]
Jul 20 09:57:29.789766 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.789738    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:29.789838 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:29.789820    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:29.828976 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:29.828925    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" podStartSLOduration=10.640864844 podStartE2EDuration="29.828911788s" totalImagesPullingTime="19.188046944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:29.827487517 +0000 UTC m=+29.840589445" watchObservedRunningTime="2026-07-20 09:57:29.828911788 +0000 UTC m=+29.842013683"
Jul 20 09:57:30.715816 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:30.715791    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 20 09:57:31.514137 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:31.514101    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:31.514137 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:31.514132    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:31.514392 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:31.514231    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:31.514392 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:31.514275    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:31.514459 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:31.514383    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:31.514490 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:31.514467    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:31.710991 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:31.710960    2572 generic.go:356] "Generic (PLEG): container finished" podID="72ae4625-f4be-43b5-ab3e-2b10725394da" containerID="338276f0cdce3675e9ae7e97383d88f38a354a956a47b86a15902004dead1ed2" exitCode=0
Jul 20 09:57:31.711149 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:31.711042    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerDied","Data":"338276f0cdce3675e9ae7e97383d88f38a354a956a47b86a15902004dead1ed2"}
Jul 20 09:57:31.711149 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:31.711130    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 20 09:57:32.307931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:32.307889    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:57:32.325843 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:32.325797    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" podUID="f4785433-f502-4748-a0d9-c718625cdd56" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 20 09:57:32.335666 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:32.335629    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" podUID="f4785433-f502-4748-a0d9-c718625cdd56" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 20 09:57:32.725007 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:32.724969    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm" podUID="f4785433-f502-4748-a0d9-c718625cdd56" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 20 09:57:33.514805 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:33.514780    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:33.515373 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:33.514807    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:33.515373 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:33.514893    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:33.515373 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:33.514902    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82lt6" podUID="f29da995-4bb0-47b2-9bd5-943406c9c977"
Jul 20 09:57:33.515373 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:33.515015    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ztdl" podUID="1e0149b6-9f1d-43fd-bca3-174fbee60711"
Jul 20 09:57:33.515373 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:33.515110    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n46hj" podUID="3700312c-9a42-4a43-a2e8-a7827283c09b"
Jul 20 09:57:34.117072 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:34.116980    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:34.117226 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:34.117168    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:34.117272 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:34.117249    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs podName:f29da995-4bb0-47b2-9bd5-943406c9c977 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:06.117227427 +0000 UTC m=+66.130329306 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs") pod "network-metrics-daemon-82lt6" (UID: "f29da995-4bb0-47b2-9bd5-943406c9c977") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:34.218363 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:34.218326    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:34.218521 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:34.218494    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:34.218603 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:34.218593    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret podName:3700312c-9a42-4a43-a2e8-a7827283c09b nodeName:}" failed. No retries permitted until 2026-07-20 09:58:06.218569854 +0000 UTC m=+66.231671742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret") pod "global-pull-secret-syncer-n46hj" (UID: "3700312c-9a42-4a43-a2e8-a7827283c09b") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:34.319220 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:34.319183    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:34.319386 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:34.319346    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:34.319386 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:34.319373    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:34.319386 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:34.319387    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pclxh for pod openshift-network-diagnostics/network-check-target-9ztdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:34.319536 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:34.319451    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh podName:1e0149b6-9f1d-43fd-bca3-174fbee60711 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:06.319433803 +0000 UTC m=+66.332535700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclxh" (UniqueName: "kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh") pod "network-check-target-9ztdl" (UID: "1e0149b6-9f1d-43fd-bca3-174fbee60711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:35.263456 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.263384    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-196.ec2.internal" event="NodeReady"
Jul 20 09:57:35.264039 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.263520    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 20 09:57:35.306332 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.306299    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s686d"]
Jul 20 09:57:35.309059 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.309024    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kglbm"]
Jul 20 09:57:35.309186 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.309154    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.310869 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.310847    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kglbm"
Jul 20 09:57:35.311927 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.311909    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 20 09:57:35.312204 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.312188    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 20 09:57:35.312290 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.312204    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9txk\""
Jul 20 09:57:35.313394 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.313245    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 20 09:57:35.313394 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.313386    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 20 09:57:35.314294 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.314275    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 20 09:57:35.314403 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.314342    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n6zqs\""
Jul 20 09:57:35.322864 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.322806    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s686d"]
Jul 20 09:57:35.324580 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.324532    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kglbm"]
Jul 20 09:57:35.335675 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.335651    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fqs2h"]
Jul 20 09:57:35.337794 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.337769    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.344008 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.343822    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 20 09:57:35.344157 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.344076    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 20 09:57:35.344157 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.344081    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 20 09:57:35.344412 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.344390    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 20 09:57:35.344515 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.344401    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m8wfj\""
Jul 20 09:57:35.357117 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.357090    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fqs2h"]
Jul 20 09:57:35.425446 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425414    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kqx\" (UniqueName: \"kubernetes.io/projected/1dbf57eb-aeb4-4414-bd2d-2c263fed7126-kube-api-access-b6kqx\") pod \"ingress-canary-kglbm\" (UID: \"1dbf57eb-aeb4-4414-bd2d-2c263fed7126\") " pod="openshift-ingress-canary/ingress-canary-kglbm"
Jul 20 09:57:35.425639 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425461    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.425639 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425499    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfc7z\" (UniqueName: \"kubernetes.io/projected/a80acdcd-5608-4ba8-bc25-82c606e1751c-kube-api-access-jfc7z\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.425639 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbf57eb-aeb4-4414-bd2d-2c263fed7126-cert\") pod \"ingress-canary-kglbm\" (UID: \"1dbf57eb-aeb4-4414-bd2d-2c263fed7126\") " pod="openshift-ingress-canary/ingress-canary-kglbm"
Jul 20 09:57:35.425639 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-data-volume\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.425639 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh65g\" (UniqueName: \"kubernetes.io/projected/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-api-access-qh65g\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.425933 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425761    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80acdcd-5608-4ba8-bc25-82c606e1751c-tmp-dir\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.425933 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425806    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.425933 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425833    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a80acdcd-5608-4ba8-bc25-82c606e1751c-config-volume\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.425933 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425856    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-crio-socket\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.425933 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.425893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.514908 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.514833    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:57:35.514908 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.514880    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:57:35.515109 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.514937    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:57:35.518765 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.518742    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27wd7\""
Jul 20 09:57:35.518894 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.518788    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:57:35.518894 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.518792    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:57:35.518894 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.518808    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x8bv\""
Jul 20 09:57:35.519211 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.519064    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:57:35.519211 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.519091    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:57:35.527219 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527199    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80acdcd-5608-4ba8-bc25-82c606e1751c-tmp-dir\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.527324 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527238    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.527324 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527266    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a80acdcd-5608-4ba8-bc25-82c606e1751c-config-volume\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.527324 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527288    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-crio-socket\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.527324 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527314    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.527530 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:35.527360    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:35.527530 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:35.527440    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:36.027421658 +0000 UTC m=+36.040523535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:35.527530 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527505    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a80acdcd-5608-4ba8-bc25-82c606e1751c-tmp-dir\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.527712 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kqx\" (UniqueName: \"kubernetes.io/projected/1dbf57eb-aeb4-4414-bd2d-2c263fed7126-kube-api-access-b6kqx\") pod \"ingress-canary-kglbm\" (UID: \"1dbf57eb-aeb4-4414-bd2d-2c263fed7126\") " pod="openshift-ingress-canary/ingress-canary-kglbm"
Jul 20 09:57:35.527712 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.527712 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527633    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfc7z\" (UniqueName: \"kubernetes.io/projected/a80acdcd-5608-4ba8-bc25-82c606e1751c-kube-api-access-jfc7z\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.527712 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527692    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbf57eb-aeb4-4414-bd2d-2c263fed7126-cert\") pod \"ingress-canary-kglbm\" (UID: \"1dbf57eb-aeb4-4414-bd2d-2c263fed7126\") " pod="openshift-ingress-canary/ingress-canary-kglbm"
Jul 20 09:57:35.527847 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-data-volume\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.527847 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:35.527720    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:35.527847 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:35.527793    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:36.027782058 +0000 UTC m=+36.040883933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:35.527847 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527794    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-crio-socket\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.527968 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:35.527903    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:35.527968 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.527917    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh65g\" (UniqueName: \"kubernetes.io/projected/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-api-access-qh65g\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.527968 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:35.527952    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls podName:a80acdcd-5608-4ba8-bc25-82c606e1751c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:36.027940561 +0000 UTC m=+36.041042435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls") pod "dns-default-s686d" (UID: "a80acdcd-5608-4ba8-bc25-82c606e1751c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:35.528198 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.528176    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-data-volume\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.528408 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.528384    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a80acdcd-5608-4ba8-bc25-82c606e1751c-config-volume\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.532583 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.532547    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbf57eb-aeb4-4414-bd2d-2c263fed7126-cert\") pod \"ingress-canary-kglbm\" (UID: \"1dbf57eb-aeb4-4414-bd2d-2c263fed7126\") " pod="openshift-ingress-canary/ingress-canary-kglbm"
Jul 20 09:57:35.543990 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.543972    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh65g\" (UniqueName: \"kubernetes.io/projected/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-api-access-qh65g\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:35.545180 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.545156    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfc7z\" (UniqueName: \"kubernetes.io/projected/a80acdcd-5608-4ba8-bc25-82c606e1751c-kube-api-access-jfc7z\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:35.545331 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.545316    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kqx\" (UniqueName: \"kubernetes.io/projected/1dbf57eb-aeb4-4414-bd2d-2c263fed7126-kube-api-access-b6kqx\") pod \"ingress-canary-kglbm\" (UID: \"1dbf57eb-aeb4-4414-bd2d-2c263fed7126\") " pod="openshift-ingress-canary/ingress-canary-kglbm"
Jul 20 09:57:35.628170 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.628135    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kglbm"
Jul 20 09:57:35.792755 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:35.792608    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kglbm"]
Jul 20 09:57:35.796245 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:35.796222    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbf57eb_aeb4_4414_bd2d_2c263fed7126.slice/crio-fe3138e423de425ada1a16902dd521fc57952bdee81a0acc3277e620fef1208b WatchSource:0}: Error finding container fe3138e423de425ada1a16902dd521fc57952bdee81a0acc3277e620fef1208b: Status 404 returned error can't find the container with id fe3138e423de425ada1a16902dd521fc57952bdee81a0acc3277e620fef1208b
Jul 20 09:57:36.032454 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.032367    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:36.032454 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.032408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:36.032677 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.032460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:36.032677 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:36.032495    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:36.032677 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:36.032580    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:36.032677 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:36.032592    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.032550558 +0000 UTC m=+37.045652437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:36.032677 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:36.032610    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:36.032677 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:36.032615    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls podName:a80acdcd-5608-4ba8-bc25-82c606e1751c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.032605849 +0000 UTC m=+37.045707724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls") pod "dns-default-s686d" (UID: "a80acdcd-5608-4ba8-bc25-82c606e1751c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:36.032677 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:36.032673    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.032659885 +0000 UTC m=+37.045761762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:36.114875 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.114846    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2dxpj_0991b138-7f5a-47b5-9fa9-436a20cc6b6c/dns-node-resolver/0.log"
Jul 20 09:57:36.616841 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.616804    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-l5j47"]
Jul 20 09:57:36.618469 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.618453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.623012 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.622989    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 20 09:57:36.623141 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.623104    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-znc4k\""
Jul 20 09:57:36.624251 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.624230    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 20 09:57:36.624354 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.624230    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 20 09:57:36.624354 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.624292    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 20 09:57:36.630202 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.630181    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-l5j47"]
Jul 20 09:57:36.727194 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.727157    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kglbm" event={"ID":"1dbf57eb-aeb4-4414-bd2d-2c263fed7126","Type":"ContainerStarted","Data":"fe3138e423de425ada1a16902dd521fc57952bdee81a0acc3277e620fef1208b"}
Jul 20 09:57:36.730687 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.730656    2572 generic.go:356] "Generic (PLEG): container finished" podID="72ae4625-f4be-43b5-ab3e-2b10725394da" containerID="7061dfaf43c620b15ec439fe3cd889fe5ff0cb6218fcd0ebde6b4b213072111d" exitCode=0
Jul 20 09:57:36.730837 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.730714    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerDied","Data":"7061dfaf43c620b15ec439fe3cd889fe5ff0cb6218fcd0ebde6b4b213072111d"}
Jul 20 09:57:36.736929 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.736905    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp4dw\" (UniqueName: \"kubernetes.io/projected/06b1649d-2168-4e86-9967-77c10b5b454d-kube-api-access-sp4dw\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.737072 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.736955    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06b1649d-2168-4e86-9967-77c10b5b454d-signing-cabundle\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.737072 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.737059    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06b1649d-2168-4e86-9967-77c10b5b454d-signing-key\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.838410 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.838222    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp4dw\" (UniqueName: \"kubernetes.io/projected/06b1649d-2168-4e86-9967-77c10b5b454d-kube-api-access-sp4dw\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.838506 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.838460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06b1649d-2168-4e86-9967-77c10b5b454d-signing-cabundle\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.838570 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.838537    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06b1649d-2168-4e86-9967-77c10b5b454d-signing-key\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.839249 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.839208    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06b1649d-2168-4e86-9967-77c10b5b454d-signing-cabundle\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.842902 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.842880    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06b1649d-2168-4e86-9967-77c10b5b454d-signing-key\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.854195 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.854171    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp4dw\" (UniqueName: \"kubernetes.io/projected/06b1649d-2168-4e86-9967-77c10b5b454d-kube-api-access-sp4dw\") pod \"service-ca-58598695f7-l5j47\" (UID: \"06b1649d-2168-4e86-9967-77c10b5b454d\") " pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:36.929080 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:36.928967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-l5j47"
Jul 20 09:57:37.040739 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:37.040527    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:37.040867 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:37.040762    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:37.040867 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:37.040804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:37.040867 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:37.040690    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:37.040986 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:37.040915    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:37.040986 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:37.040972    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:37.040986 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:37.040982    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls podName:a80acdcd-5608-4ba8-bc25-82c606e1751c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:39.040959486 +0000 UTC m=+39.054061361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls") pod "dns-default-s686d" (UID: "a80acdcd-5608-4ba8-bc25-82c606e1751c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:37.041110 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:37.041025    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:39.041005719 +0000 UTC m=+39.054107604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:37.041110 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:37.041072    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:39.041059828 +0000 UTC m=+39.054161702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:37.077857 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:37.077802    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-l5j47"]
Jul 20 09:57:37.082768 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:37.082740    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b1649d_2168_4e86_9967_77c10b5b454d.slice/crio-8e254ea0c97d117bfba142a08b23470bbb18709eb2ff3980793d89516f6edc05 WatchSource:0}: Error finding container 8e254ea0c97d117bfba142a08b23470bbb18709eb2ff3980793d89516f6edc05: Status 404 returned error can't find the container with id 8e254ea0c97d117bfba142a08b23470bbb18709eb2ff3980793d89516f6edc05
Jul 20 09:57:37.087799 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:37.087781    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-66rz6_c443edf8-fe49-4b79-ae03-a78c32a1b5af/node-ca/0.log"
Jul 20 09:57:37.735741 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:37.735671    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-l5j47" event={"ID":"06b1649d-2168-4e86-9967-77c10b5b454d","Type":"ContainerStarted","Data":"8e254ea0c97d117bfba142a08b23470bbb18709eb2ff3980793d89516f6edc05"}
Jul 20 09:57:37.739824 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:37.739669    2572 generic.go:356] "Generic (PLEG): container finished" podID="72ae4625-f4be-43b5-ab3e-2b10725394da" containerID="159342da175cca339a6bae5b8583bc75f57ee545aa3a9ce90fb3fda88064050a" exitCode=0
Jul 20 09:57:37.739824 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:37.739728    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerDied","Data":"159342da175cca339a6bae5b8583bc75f57ee545aa3a9ce90fb3fda88064050a"}
Jul 20 09:57:39.058281 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.058250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:39.058817 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.058313    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:39.058817 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.058335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:39.058817 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:39.058401    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:39.058817 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:39.058415    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:39.058817 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:39.058427    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:39.058817 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:39.058463    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:43.058450705 +0000 UTC m=+43.071552579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:39.058817 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:39.058477    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:43.058470135 +0000 UTC m=+43.071572009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:39.058817 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:39.058487    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls podName:a80acdcd-5608-4ba8-bc25-82c606e1751c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:43.058481379 +0000 UTC m=+43.071583252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls") pod "dns-default-s686d" (UID: "a80acdcd-5608-4ba8-bc25-82c606e1751c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:39.746089 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.746051    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kglbm" event={"ID":"1dbf57eb-aeb4-4414-bd2d-2c263fed7126","Type":"ContainerStarted","Data":"133a01f8a9a5f997340ce35274d0387a6189fc832e256eba1e3315c59180e238"}
Jul 20 09:57:39.747478 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.747449    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-l5j47" event={"ID":"06b1649d-2168-4e86-9967-77c10b5b454d","Type":"ContainerStarted","Data":"2f5d88e1d9e18687409c65c8cb3c0f3d2f20bd5490bf2870727585e9a2e74eed"}
Jul 20 09:57:39.750881 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.750860    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zp58" event={"ID":"72ae4625-f4be-43b5-ab3e-2b10725394da","Type":"ContainerStarted","Data":"d1f75e2094f023211556d2cd2f15aad0f2dbb52fb1f810d8441dc67bb1165706"}
Jul 20 09:57:39.791664 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.791621    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kglbm" podStartSLOduration=1.476346521 podStartE2EDuration="4.791606288s" totalImagesPullingTime="3.315259767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:39.789669875 +0000 UTC m=+39.802771771" watchObservedRunningTime="2026-07-20 09:57:39.791606288 +0000 UTC m=+39.804708162"
Jul 20 09:57:39.829714 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.829659    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6zp58" podStartSLOduration=11.091267002 podStartE2EDuration="39.829643369s" totalImagesPullingTime="28.738376367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:39.827965531 +0000 UTC m=+39.841067418" watchObservedRunningTime="2026-07-20 09:57:39.829643369 +0000 UTC m=+39.842745267"
Jul 20 09:57:39.849828 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:39.849783    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-l5j47" podStartSLOduration=1.815709981 podStartE2EDuration="3.84976989s" totalImagesPullingTime="2.034059909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:39.84907596 +0000 UTC m=+39.862177855" watchObservedRunningTime="2026-07-20 09:57:39.84976989 +0000 UTC m=+39.862871786"
Jul 20 09:57:43.092521 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:43.092480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:43.093183 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:43.092542    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:43.093183 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:43.092633    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:43.093183 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:43.092649    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:43.093183 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:43.092699    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:43.093183 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:43.092711    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:51.092694602 +0000 UTC m=+51.105796476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:43.093183 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:43.092716    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:43.093183 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:43.092746    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm podName:21a853c5-75ff-4f78-8c9d-ad305c4d1a1d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:51.092733452 +0000 UTC m=+51.105835341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fqs2h" (UID: "21a853c5-75ff-4f78-8c9d-ad305c4d1a1d") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:43.093183 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:57:43.092768    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls podName:a80acdcd-5608-4ba8-bc25-82c606e1751c nodeName:}" failed. No retries permitted until 2026-07-20 09:57:51.092752417 +0000 UTC m=+51.105854296 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls") pod "dns-default-s686d" (UID: "a80acdcd-5608-4ba8-bc25-82c606e1751c") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:51.148857 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.148820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:51.148857 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.148861    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:51.149356 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.148895    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:51.149395 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.149379    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:51.152221 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.152194    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80acdcd-5608-4ba8-bc25-82c606e1751c-metrics-tls\") pod \"dns-default-s686d\" (UID: \"a80acdcd-5608-4ba8-bc25-82c606e1751c\") " pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:51.152334 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.152297    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21a853c5-75ff-4f78-8c9d-ad305c4d1a1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqs2h\" (UID: \"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d\") " pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:51.222362 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.222326    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:51.247382 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.247363    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fqs2h"
Jul 20 09:57:51.372858 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.372825    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s686d"]
Jul 20 09:57:51.375949 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:51.375907    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80acdcd_5608_4ba8_bc25_82c606e1751c.slice/crio-f601ed9737a04a59f94e1ce7513c5a92ab3f059dca151e91aee605432d2891b8 WatchSource:0}: Error finding container f601ed9737a04a59f94e1ce7513c5a92ab3f059dca151e91aee605432d2891b8: Status 404 returned error can't find the container with id f601ed9737a04a59f94e1ce7513c5a92ab3f059dca151e91aee605432d2891b8
Jul 20 09:57:51.399506 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.399452    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fqs2h"]
Jul 20 09:57:51.402796 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:57:51.402766    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a853c5_75ff_4f78_8c9d_ad305c4d1a1d.slice/crio-7d3d4ff7c42760eb8b3ec7a704c1fc26e16e0df78d4902c0909d4afa157cd952 WatchSource:0}: Error finding container 7d3d4ff7c42760eb8b3ec7a704c1fc26e16e0df78d4902c0909d4afa157cd952: Status 404 returned error can't find the container with id 7d3d4ff7c42760eb8b3ec7a704c1fc26e16e0df78d4902c0909d4afa157cd952
Jul 20 09:57:51.780525 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.780489    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fqs2h" event={"ID":"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d","Type":"ContainerStarted","Data":"7d3d4ff7c42760eb8b3ec7a704c1fc26e16e0df78d4902c0909d4afa157cd952"}
Jul 20 09:57:51.781609 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:51.781585    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s686d" event={"ID":"a80acdcd-5608-4ba8-bc25-82c606e1751c","Type":"ContainerStarted","Data":"f601ed9737a04a59f94e1ce7513c5a92ab3f059dca151e91aee605432d2891b8"}
Jul 20 09:57:56.799610 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:56.799569    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fqs2h" event={"ID":"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d","Type":"ContainerStarted","Data":"75cd1932b7e5f37cf72fcad2782b3c5c014bf44cebfd7c8e54eda60257f02567"}
Jul 20 09:57:56.801330 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:56.801302    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s686d" event={"ID":"a80acdcd-5608-4ba8-bc25-82c606e1751c","Type":"ContainerStarted","Data":"bb3dd62371d37903564a659e8c57196967240c73663738ac889fab6f9485fd55"}
Jul 20 09:57:57.807287 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:57.807244    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s686d" event={"ID":"a80acdcd-5608-4ba8-bc25-82c606e1751c","Type":"ContainerStarted","Data":"b07f3e9cce3656839f2ed3f4619d5271690a65e2cc90e57aec8485995fa31c76"}
Jul 20 09:57:57.807781 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:57.807406    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s686d"
Jul 20 09:57:57.826698 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:57:57.826637    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s686d" podStartSLOduration=17.544041911 podStartE2EDuration="22.826615847s" totalImagesPullingTime="5.282573936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:57.825345542 +0000 UTC m=+57.838447439" watchObservedRunningTime="2026-07-20 09:57:57.826615847 +0000 UTC m=+57.839717741"
Jul 20 09:58:00.058220 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.058191    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"]
Jul 20 09:58:00.074440 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.074411    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"
Jul 20 09:58:00.077599 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.077547    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fvsd9\""
Jul 20 09:58:00.078931 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.078899    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 20 09:58:00.079093 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.078900    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 20 09:58:00.080062 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.080031    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"]
Jul 20 09:58:00.091822 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.091800    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"]
Jul 20 09:58:00.110486 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.110456    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"]
Jul 20 09:58:00.110486 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.110475    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.113574 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.113536    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 20 09:58:00.114179 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.114155    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 20 09:58:00.114397 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.114381    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 20 09:58:00.114524 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.114505    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 20 09:58:00.120380 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.120356    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59b0e12e-76ad-4a85-b34a-46f765f89fa8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6zs8d\" (UID: \"59b0e12e-76ad-4a85-b34a-46f765f89fa8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"
Jul 20 09:58:00.120493 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.120393    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59b0e12e-76ad-4a85-b34a-46f765f89fa8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6zs8d\" (UID: \"59b0e12e-76ad-4a85-b34a-46f765f89fa8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"
Jul 20 09:58:00.125083 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.125061    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"
Jul 20 09:58:00.131776 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.131749    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"]
Jul 20 09:58:00.132520 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.132501    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4tpbg\""
Jul 20 09:58:00.132800 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.132780    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 20 09:58:00.133839 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.133819    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"]
Jul 20 09:58:00.174929 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.174907    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c56464c7b-b9pvp"]
Jul 20 09:58:00.194639 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.194619    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-8n9sc"]
Jul 20 09:58:00.194775 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.194748    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.199096 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.198895    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 20 09:58:00.199096 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.198992    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x8ldn\""
Jul 20 09:58:00.199257 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.199145    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 20 09:58:00.199684 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.199660    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 20 09:58:00.204582 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.204539    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 20 09:58:00.218782 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.218763    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-8n9sc"]
Jul 20 09:58:00.218782 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.218789    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c56464c7b-b9pvp"]
Jul 20 09:58:00.218917 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.218863    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-8n9sc"
Jul 20 09:58:00.220988 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.220967    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59b0e12e-76ad-4a85-b34a-46f765f89fa8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6zs8d\" (UID: \"59b0e12e-76ad-4a85-b34a-46f765f89fa8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"
Jul 20 09:58:00.221109 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.220999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59b0e12e-76ad-4a85-b34a-46f765f89fa8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6zs8d\" (UID: \"59b0e12e-76ad-4a85-b34a-46f765f89fa8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"
Jul 20 09:58:00.221109 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221045    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzcbx\" (UniqueName: \"kubernetes.io/projected/902e58f7-6c5b-46c0-97b2-117452e776f8-kube-api-access-vzcbx\") pod \"managed-serviceaccount-addon-agent-58f49644f6-wd9db\" (UID: \"902e58f7-6c5b-46c0-97b2-117452e776f8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"
Jul 20 09:58:00.221109 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221075    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-registry-certificates\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.221109 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-tmp\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.221503 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221143    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvs8\" (UniqueName: \"kubernetes.io/projected/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-kube-api-access-ftvs8\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.221503 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221188    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-image-registry-private-configuration\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.221503 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221226    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-registry-tls\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.221503 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/902e58f7-6c5b-46c0-97b2-117452e776f8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58f49644f6-wd9db\" (UID: \"902e58f7-6c5b-46c0-97b2-117452e776f8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"
Jul 20 09:58:00.221503 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221279    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-ca-trust-extracted\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.221503 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221309    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-bound-sa-token\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.221503 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221428    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.221880 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221510    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-installation-pull-secrets\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.221880 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p2kv\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-kube-api-access-6p2kv\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.221880 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221588    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 20 09:58:00.221880 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-trusted-ca\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.221880 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221645    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59b0e12e-76ad-4a85-b34a-46f765f89fa8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6zs8d\" (UID: \"59b0e12e-76ad-4a85-b34a-46f765f89fa8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"
Jul 20 09:58:00.221880 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221821    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 20 09:58:00.222108 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.221445    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-k5r8b\""
Jul 20 09:58:00.223591 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.223570    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59b0e12e-76ad-4a85-b34a-46f765f89fa8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6zs8d\" (UID: \"59b0e12e-76ad-4a85-b34a-46f765f89fa8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"
Jul 20 09:58:00.322824 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.322759    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzcbx\" (UniqueName: \"kubernetes.io/projected/902e58f7-6c5b-46c0-97b2-117452e776f8-kube-api-access-vzcbx\") pod \"managed-serviceaccount-addon-agent-58f49644f6-wd9db\" (UID: \"902e58f7-6c5b-46c0-97b2-117452e776f8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"
Jul 20 09:58:00.322824 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.322804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-registry-certificates\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.322824 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.322820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-tmp\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.323091 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.322848    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvs8\" (UniqueName: \"kubernetes.io/projected/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-kube-api-access-ftvs8\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.323091 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.322872    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-image-registry-private-configuration\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.323091 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.322898    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-registry-tls\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.323091 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.322922    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/902e58f7-6c5b-46c0-97b2-117452e776f8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58f49644f6-wd9db\" (UID: \"902e58f7-6c5b-46c0-97b2-117452e776f8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"
Jul 20 09:58:00.323091 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323018    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-ca-trust-extracted\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.323091 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323058    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-bound-sa-token\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.323547 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323108    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.323547 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323159    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddtpb\" (UniqueName: \"kubernetes.io/projected/cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a-kube-api-access-ddtpb\") pod \"downloads-5bc67dcd46-8n9sc\" (UID: \"cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a\") " pod="openshift-console/downloads-5bc67dcd46-8n9sc"
Jul 20 09:58:00.323547 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-installation-pull-secrets\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.323547 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p2kv\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-kube-api-access-6p2kv\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.323547 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323261    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-tmp\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.323547 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323279    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-trusted-ca\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.323547 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323438    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-ca-trust-extracted\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.323976 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.323768    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-registry-certificates\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.324290 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.324263    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-trusted-ca\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.325760 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.325730    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/902e58f7-6c5b-46c0-97b2-117452e776f8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58f49644f6-wd9db\" (UID: \"902e58f7-6c5b-46c0-97b2-117452e776f8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"
Jul 20 09:58:00.325874 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.325764    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-installation-pull-secrets\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.325936 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.325907    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-image-registry-private-configuration\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.326085 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.326066    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-registry-tls\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.326166 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.326149    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.341284 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.341267    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 20 09:58:00.343230 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.343211    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-bound-sa-token\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.344570 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.344538    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p2kv\" (UniqueName: \"kubernetes.io/projected/6d129d25-6c55-47d1-8fa1-d83bdd5540ba-kube-api-access-6p2kv\") pod \"image-registry-6c56464c7b-b9pvp\" (UID: \"6d129d25-6c55-47d1-8fa1-d83bdd5540ba\") " pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.345624 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.345606    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 20 09:58:00.353535 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.353513    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzcbx\" (UniqueName: \"kubernetes.io/projected/902e58f7-6c5b-46c0-97b2-117452e776f8-kube-api-access-vzcbx\") pod \"managed-serviceaccount-addon-agent-58f49644f6-wd9db\" (UID: \"902e58f7-6c5b-46c0-97b2-117452e776f8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"
Jul 20 09:58:00.355411 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.355392    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvs8\" (UniqueName: \"kubernetes.io/projected/cd8ce203-f2ef-4311-bf2e-3473ed1d925b-kube-api-access-ftvs8\") pod \"klusterlet-addon-workmgr-7445498dd5-hpngt\" (UID: \"cd8ce203-f2ef-4311-bf2e-3473ed1d925b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.387157 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.387137    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fvsd9\""
Jul 20 09:58:00.395572 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.395539    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"
Jul 20 09:58:00.423782 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.423756    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddtpb\" (UniqueName: \"kubernetes.io/projected/cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a-kube-api-access-ddtpb\") pod \"downloads-5bc67dcd46-8n9sc\" (UID: \"cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a\") " pod="openshift-console/downloads-5bc67dcd46-8n9sc"
Jul 20 09:58:00.423919 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.423904    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 20 09:58:00.431532 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.431515    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:00.437805 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.437789    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 20 09:58:00.444006 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.443992    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4tpbg\""
Jul 20 09:58:00.448760 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.448745    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 20 09:58:00.451700 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.451680    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"
Jul 20 09:58:00.459115 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.459091    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddtpb\" (UniqueName: \"kubernetes.io/projected/cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a-kube-api-access-ddtpb\") pod \"downloads-5bc67dcd46-8n9sc\" (UID: \"cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a\") " pod="openshift-console/downloads-5bc67dcd46-8n9sc"
Jul 20 09:58:00.509942 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.509918    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x8ldn\""
Jul 20 09:58:00.517504 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.517473    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:00.536935 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.536919    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-k5r8b\""
Jul 20 09:58:00.544003 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.543987    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-8n9sc"
Jul 20 09:58:00.820599 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.820459    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"]
Jul 20 09:58:00.821267 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:00.821192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fqs2h" event={"ID":"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d","Type":"ContainerStarted","Data":"936f87cc148a9995bc8d9f0613e144f72ec094b1ce9cf85e590f601c0cc6da09"}
Jul 20 09:58:01.047046 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.047015    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db"]
Jul 20 09:58:01.049542 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:01.049518    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902e58f7_6c5b_46c0_97b2_117452e776f8.slice/crio-9381cc3217f047c6d7a8fad2304576bb118ea40c470e6beb71b1c49a5e0cc913 WatchSource:0}: Error finding container 9381cc3217f047c6d7a8fad2304576bb118ea40c470e6beb71b1c49a5e0cc913: Status 404 returned error can't find the container with id 9381cc3217f047c6d7a8fad2304576bb118ea40c470e6beb71b1c49a5e0cc913
Jul 20 09:58:01.052129 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.052109    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d"]
Jul 20 09:58:01.055786 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.055764    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-8n9sc"]
Jul 20 09:58:01.056488 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:01.056443    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b0e12e_76ad_4a85_b34a_46f765f89fa8.slice/crio-840df49bcc735e74a3e2455c1f29723e1319a0e0aff7ad319e4e847595c6455d WatchSource:0}: Error finding container 840df49bcc735e74a3e2455c1f29723e1319a0e0aff7ad319e4e847595c6455d: Status 404 returned error can't find the container with id 840df49bcc735e74a3e2455c1f29723e1319a0e0aff7ad319e4e847595c6455d
Jul 20 09:58:01.058307 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:01.058286    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebf0a31_f95c_4ea3_baf6_7e4b2e3fcf7a.slice/crio-0605e68dc4df5bfc15acb893da27ee682ed5764e36b601207fbc3bc6ca8e7ddd WatchSource:0}: Error finding container 0605e68dc4df5bfc15acb893da27ee682ed5764e36b601207fbc3bc6ca8e7ddd: Status 404 returned error can't find the container with id 0605e68dc4df5bfc15acb893da27ee682ed5764e36b601207fbc3bc6ca8e7ddd
Jul 20 09:58:01.061577 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.061453    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c56464c7b-b9pvp"]
Jul 20 09:58:01.063824 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:01.063789    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d129d25_6c55_47d1_8fa1_d83bdd5540ba.slice/crio-e1daf8aa3b76254cb3df75245a2da1ab5f5fcc38fc4cc530b573e00363a0e446 WatchSource:0}: Error finding container e1daf8aa3b76254cb3df75245a2da1ab5f5fcc38fc4cc530b573e00363a0e446: Status 404 returned error can't find the container with id e1daf8aa3b76254cb3df75245a2da1ab5f5fcc38fc4cc530b573e00363a0e446
Jul 20 09:58:01.827951 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.827874    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d" event={"ID":"59b0e12e-76ad-4a85-b34a-46f765f89fa8","Type":"ContainerStarted","Data":"840df49bcc735e74a3e2455c1f29723e1319a0e0aff7ad319e4e847595c6455d"}
Jul 20 09:58:01.838449 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.836295    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-8n9sc" event={"ID":"cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a","Type":"ContainerStarted","Data":"0605e68dc4df5bfc15acb893da27ee682ed5764e36b601207fbc3bc6ca8e7ddd"}
Jul 20 09:58:01.839524 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.839496    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp" event={"ID":"6d129d25-6c55-47d1-8fa1-d83bdd5540ba","Type":"ContainerStarted","Data":"237313e308098edd7fe24c58dfb21d4f1b83da77ad40b8c983fbcd35309cf691"}
Jul 20 09:58:01.839764 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.839740    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp" event={"ID":"6d129d25-6c55-47d1-8fa1-d83bdd5540ba","Type":"ContainerStarted","Data":"e1daf8aa3b76254cb3df75245a2da1ab5f5fcc38fc4cc530b573e00363a0e446"}
Jul 20 09:58:01.839902 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.839888    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:01.842512 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.842452    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt" event={"ID":"cd8ce203-f2ef-4311-bf2e-3473ed1d925b","Type":"ContainerStarted","Data":"0aaf25ed1ac1f045da3710706bd880c816e8c17cd9f1df593850d43979c9c8e8"}
Jul 20 09:58:01.846830 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.846777    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db" event={"ID":"902e58f7-6c5b-46c0-97b2-117452e776f8","Type":"ContainerStarted","Data":"9381cc3217f047c6d7a8fad2304576bb118ea40c470e6beb71b1c49a5e0cc913"}
Jul 20 09:58:01.867407 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:01.867056    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp" podStartSLOduration=1.867034173 podStartE2EDuration="1.867034173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:01.865580137 +0000 UTC m=+61.878682034" watchObservedRunningTime="2026-07-20 09:58:01.867034173 +0000 UTC m=+61.880136062"
Jul 20 09:58:02.738063 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:02.738025    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4csm"
Jul 20 09:58:03.866029 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:03.865948    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d" event={"ID":"59b0e12e-76ad-4a85-b34a-46f765f89fa8","Type":"ContainerStarted","Data":"c84326b342d13244260b1a2e4f53f3ce11cc0389aebbd4bfd589cb8e398ebb66"}
Jul 20 09:58:03.885625 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:03.884591    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6zs8d" podStartSLOduration=2.000907405 podStartE2EDuration="3.884568755s" totalImagesPullingTime="1.88366135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:03.884041768 +0000 UTC m=+63.897143665" watchObservedRunningTime="2026-07-20 09:58:03.884568755 +0000 UTC m=+63.897670645"
Jul 20 09:58:05.888032 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:05.887969    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"]
Jul 20 09:58:05.906014 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:05.905969    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"]
Jul 20 09:58:05.906188 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:05.906087    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"
Jul 20 09:58:05.911334 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:05.911309    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 20 09:58:05.911488 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:05.911468    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zmcrm\""
Jul 20 09:58:05.980982 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:05.980940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b15c8705-6e49-4c81-a40d-5d867e84f9c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-5kt4c\" (UID: \"b15c8705-6e49-4c81-a40d-5d867e84f9c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"
Jul 20 09:58:06.082345 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.082304    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b15c8705-6e49-4c81-a40d-5d867e84f9c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-5kt4c\" (UID: \"b15c8705-6e49-4c81-a40d-5d867e84f9c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"
Jul 20 09:58:06.085280 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.085251    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b15c8705-6e49-4c81-a40d-5d867e84f9c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-5kt4c\" (UID: \"b15c8705-6e49-4c81-a40d-5d867e84f9c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"
Jul 20 09:58:06.183127 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.183010    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:58:06.186154 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.186126    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:58:06.196152 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.196125    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29da995-4bb0-47b2-9bd5-943406c9c977-metrics-certs\") pod \"network-metrics-daemon-82lt6\" (UID: \"f29da995-4bb0-47b2-9bd5-943406c9c977\") " pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:58:06.218311 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.218172    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"
Jul 20 09:58:06.284386 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.284328    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:58:06.287669 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.287640    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:58:06.298015 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.297954    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3700312c-9a42-4a43-a2e8-a7827283c09b-original-pull-secret\") pod \"global-pull-secret-syncer-n46hj\" (UID: \"3700312c-9a42-4a43-a2e8-a7827283c09b\") " pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:58:06.384949 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.384916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:58:06.388126 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.388097    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:58:06.398142 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.398118    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:58:06.409147 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.409118    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pclxh\" (UniqueName: \"kubernetes.io/projected/1e0149b6-9f1d-43fd-bca3-174fbee60711-kube-api-access-pclxh\") pod \"network-check-target-9ztdl\" (UID: \"1e0149b6-9f1d-43fd-bca3-174fbee60711\") " pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:58:06.428305 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.428277    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27wd7\""
Jul 20 09:58:06.435753 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.435691    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:58:06.435973 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.435958    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x8bv\""
Jul 20 09:58:06.444280 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.444258    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82lt6"
Jul 20 09:58:06.444401 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:06.444373    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n46hj"
Jul 20 09:58:07.813173 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:07.813142    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s686d"
Jul 20 09:58:08.729799 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.727794    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"]
Jul 20 09:58:08.733294 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:08.732835    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15c8705_6e49_4c81_a40d_5d867e84f9c3.slice/crio-8e378ab92d73ac6766e48240dd2e165be4c5c0545cf0fd36c87d1ae21eb847d0 WatchSource:0}: Error finding container 8e378ab92d73ac6766e48240dd2e165be4c5c0545cf0fd36c87d1ae21eb847d0: Status 404 returned error can't find the container with id 8e378ab92d73ac6766e48240dd2e165be4c5c0545cf0fd36c87d1ae21eb847d0
Jul 20 09:58:08.894463 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.894371    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt" event={"ID":"cd8ce203-f2ef-4311-bf2e-3473ed1d925b","Type":"ContainerStarted","Data":"a0fdbe2a2c222feb9c204b2544f5214077a1eb960884d324c0a4588d745a720d"}
Jul 20 09:58:08.894928 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.894597    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:08.896161 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.896106    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db" event={"ID":"902e58f7-6c5b-46c0-97b2-117452e776f8","Type":"ContainerStarted","Data":"fa86be3efef365de2ec522e26b32ee0f4496968f513a769175a3b92e9935e5bc"}
Jul 20 09:58:08.896708 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.896688    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt"
Jul 20 09:58:08.897687 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.897662    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c" event={"ID":"b15c8705-6e49-4c81-a40d-5d867e84f9c3","Type":"ContainerStarted","Data":"8e378ab92d73ac6766e48240dd2e165be4c5c0545cf0fd36c87d1ae21eb847d0"}
Jul 20 09:58:08.900049 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.900023    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fqs2h" event={"ID":"21a853c5-75ff-4f78-8c9d-ad305c4d1a1d","Type":"ContainerStarted","Data":"1182247fb537a7cd494ddac916113013238097b1afb206cfaefd35fe3c501354"}
Jul 20 09:58:08.915882 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.915831    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445498dd5-hpngt" podStartSLOduration=1.182265701 podStartE2EDuration="8.915813102s" totalImagesPullingTime="7.733547401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:08.915639425 +0000 UTC m=+68.928741323" watchObservedRunningTime="2026-07-20 09:58:08.915813102 +0000 UTC m=+68.928915001"
Jul 20 09:58:08.942464 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.942391    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fqs2h" podStartSLOduration=17.024586566 podStartE2EDuration="33.942367817s" totalImagesPullingTime="16.917781251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:08.940209982 +0000 UTC m=+68.953311879" watchObservedRunningTime="2026-07-20 09:58:08.942367817 +0000 UTC m=+68.955469716"
Jul 20 09:58:08.956571 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.956507    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58f49644f6-wd9db" podStartSLOduration=1.477506018 podStartE2EDuration="8.956491111s" totalImagesPullingTime="7.478985093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:08.955808199 +0000 UTC m=+68.968910096" watchObservedRunningTime="2026-07-20 09:58:08.956491111 +0000 UTC m=+68.969593008"
Jul 20 09:58:08.964274 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.964243    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-82lt6"]
Jul 20 09:58:08.965310 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.965284    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n46hj"]
Jul 20 09:58:08.967177 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:08.967152    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29da995_4bb0_47b2_9bd5_943406c9c977.slice/crio-a584e6c590579b8373758ccac3b3e77fc93c97e754ffe75a1718216d7a33bada WatchSource:0}: Error finding container a584e6c590579b8373758ccac3b3e77fc93c97e754ffe75a1718216d7a33bada: Status 404 returned error can't find the container with id a584e6c590579b8373758ccac3b3e77fc93c97e754ffe75a1718216d7a33bada
Jul 20 09:58:08.967971 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:08.967887    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3700312c_9a42_4a43_a2e8_a7827283c09b.slice/crio-499d87ccd2ae818f8de7bd986667bafa83acbe8e58afe067d8764a4216ce43a9 WatchSource:0}: Error finding container 499d87ccd2ae818f8de7bd986667bafa83acbe8e58afe067d8764a4216ce43a9: Status 404 returned error can't find the container with id 499d87ccd2ae818f8de7bd986667bafa83acbe8e58afe067d8764a4216ce43a9
Jul 20 09:58:08.970085 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:08.970028    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9ztdl"]
Jul 20 09:58:08.972676 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:08.972651    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0149b6_9f1d_43fd_bca3_174fbee60711.slice/crio-edf5d54b00759dfb4b8d67434b1b42749d3dc93f7fce0845fe9a4f786a49896c WatchSource:0}: Error finding container edf5d54b00759dfb4b8d67434b1b42749d3dc93f7fce0845fe9a4f786a49896c: Status 404 returned error can't find the container with id edf5d54b00759dfb4b8d67434b1b42749d3dc93f7fce0845fe9a4f786a49896c
Jul 20 09:58:09.909284 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:09.909242    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9ztdl" event={"ID":"1e0149b6-9f1d-43fd-bca3-174fbee60711","Type":"ContainerStarted","Data":"edf5d54b00759dfb4b8d67434b1b42749d3dc93f7fce0845fe9a4f786a49896c"}
Jul 20 09:58:09.912310 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:09.912275    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-82lt6" event={"ID":"f29da995-4bb0-47b2-9bd5-943406c9c977","Type":"ContainerStarted","Data":"a584e6c590579b8373758ccac3b3e77fc93c97e754ffe75a1718216d7a33bada"}
Jul 20 09:58:09.917234 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:09.916107    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n46hj" event={"ID":"3700312c-9a42-4a43-a2e8-a7827283c09b","Type":"ContainerStarted","Data":"499d87ccd2ae818f8de7bd986667bafa83acbe8e58afe067d8764a4216ce43a9"}
Jul 20 09:58:11.926212 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:11.926151    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c" event={"ID":"b15c8705-6e49-4c81-a40d-5d867e84f9c3","Type":"ContainerStarted","Data":"2f08083ddfa6eccfdb335b85b8a485fc6588376a2b43e4c21f3fb0c18d0f1ac5"}
Jul 20 09:58:11.927134 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:11.926970    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"
Jul 20 09:58:11.930357 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:11.929794    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-82lt6" event={"ID":"f29da995-4bb0-47b2-9bd5-943406c9c977","Type":"ContainerStarted","Data":"e9f4c817f02677631df3b15ec3308e7d785320ee342a40a18ef853225a507bb0"}
Jul 20 09:58:11.930357 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:11.929849    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-82lt6" event={"ID":"f29da995-4bb0-47b2-9bd5-943406c9c977","Type":"ContainerStarted","Data":"5c9fdd5a2fc46cd8cfd3677aa7312aaab5a8f37e5f6f53c7a4597d9b9f8e8d5c"}
Jul 20 09:58:11.933330 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:11.933305    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c"
Jul 20 09:58:11.944165 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:11.944117    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-5kt4c" podStartSLOduration=4.632166926 podStartE2EDuration="6.944097411s" totalImagesPullingTime="2.311930485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:11.943270604 +0000 UTC m=+71.956372513" watchObservedRunningTime="2026-07-20 09:58:11.944097411 +0000 UTC m=+71.957199307"
Jul 20 09:58:11.962479 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:11.962034    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-82lt6" podStartSLOduration=69.859757414 podStartE2EDuration="1m11.962011981s" totalImagesPullingTime="2.102254567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:11.960303243 +0000 UTC m=+71.973405136" watchObservedRunningTime="2026-07-20 09:58:11.962011981 +0000 UTC m=+71.975113880"
Jul 20 09:58:12.965350 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.965099    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"]
Jul 20 09:58:12.972611 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.968985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:12.972611 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.972501    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 20 09:58:12.974997 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.974959    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 20 09:58:12.975123 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.975047    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 20 09:58:12.975606 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.975465    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 20 09:58:12.976134 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.976112    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jjh6h\""
Jul 20 09:58:12.976134 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.976133    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 20 09:58:12.979856 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:12.979833    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"]
Jul 20 09:58:13.044398 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.044355    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.044594 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.044410    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.044594 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.044469    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mvg\" (UniqueName: \"kubernetes.io/projected/c93d2f2a-5a06-4bea-a170-db5aaa435782-kube-api-access-78mvg\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.044594 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.044524    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c93d2f2a-5a06-4bea-a170-db5aaa435782-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.145727 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.145687    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.145906 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.145852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mvg\" (UniqueName: \"kubernetes.io/projected/c93d2f2a-5a06-4bea-a170-db5aaa435782-kube-api-access-78mvg\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.145990 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.145927    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c93d2f2a-5a06-4bea-a170-db5aaa435782-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.145990 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.145967    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.146080 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:58:13.146055    2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 20 09:58:13.146133 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:58:13.146111    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-tls podName:c93d2f2a-5a06-4bea-a170-db5aaa435782 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:13.646091463 +0000 UTC m=+73.659193342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-9jx8b" (UID: "c93d2f2a-5a06-4bea-a170-db5aaa435782") : secret "prometheus-operator-tls" not found
Jul 20 09:58:13.146737 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.146714    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c93d2f2a-5a06-4bea-a170-db5aaa435782-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.148280 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.148260    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.155085 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.155037    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mvg\" (UniqueName: \"kubernetes.io/projected/c93d2f2a-5a06-4bea-a170-db5aaa435782-kube-api-access-78mvg\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.649969 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.649933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.652596 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.652574    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93d2f2a-5a06-4bea-a170-db5aaa435782-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9jx8b\" (UID: \"c93d2f2a-5a06-4bea-a170-db5aaa435782\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:13.883221 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:13.883186    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"
Jul 20 09:58:21.784590 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.784534    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b"]
Jul 20 09:58:21.787315 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:21.787290    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93d2f2a_5a06_4bea_a170_db5aaa435782.slice/crio-195a8b0c35c9f2ac359cb3f9317accf22a188d7a559ebb713bee671727199134 WatchSource:0}: Error finding container 195a8b0c35c9f2ac359cb3f9317accf22a188d7a559ebb713bee671727199134: Status 404 returned error can't find the container with id 195a8b0c35c9f2ac359cb3f9317accf22a188d7a559ebb713bee671727199134
Jul 20 09:58:21.970536 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.970501    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9ztdl" event={"ID":"1e0149b6-9f1d-43fd-bca3-174fbee60711","Type":"ContainerStarted","Data":"715c96aa178258f7a122341b7ed68677fa402a0428ca44a9311d918bfe02afd8"}
Jul 20 09:58:21.970733 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.970613    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:58:21.972169 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.972110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n46hj" event={"ID":"3700312c-9a42-4a43-a2e8-a7827283c09b","Type":"ContainerStarted","Data":"27e7dd73a8f1c274a5eac44fd46a45d1f5c3c4341dd5a82c624155470e50c35e"}
Jul 20 09:58:21.973196 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.973172    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b" event={"ID":"c93d2f2a-5a06-4bea-a170-db5aaa435782","Type":"ContainerStarted","Data":"195a8b0c35c9f2ac359cb3f9317accf22a188d7a559ebb713bee671727199134"}
Jul 20 09:58:21.974467 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.974446    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-8n9sc" event={"ID":"cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a","Type":"ContainerStarted","Data":"2efa739d8354ee8f2b51a7ea75e8b9e1c8c0f5fe29fd35d674ace3aae5764c03"}
Jul 20 09:58:21.974568 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.974487    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-8n9sc"
Jul 20 09:58:21.976022 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.975999    2572 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-8n9sc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused" start-of-body=
Jul 20 09:58:21.976104 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.976043    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-8n9sc" podUID="cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused"
Jul 20 09:58:21.988841 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:21.988803    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9ztdl" podStartSLOduration=69.320501878 podStartE2EDuration="1m21.988789016s" totalImagesPullingTime="12.668287138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:21.987237083 +0000 UTC m=+82.000338978" watchObservedRunningTime="2026-07-20 09:58:21.988789016 +0000 UTC m=+82.001890944"
Jul 20 09:58:22.003019 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:22.002969    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n46hj" podStartSLOduration=68.3289245 podStartE2EDuration="1m21.002949728s" totalImagesPullingTime="12.674025228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:22.002767037 +0000 UTC m=+82.015868934" watchObservedRunningTime="2026-07-20 09:58:22.002949728 +0000 UTC m=+82.016051605"
Jul 20 09:58:22.020031 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:22.019984    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-8n9sc" podStartSLOduration=1.374113335 podStartE2EDuration="22.019970661s" totalImagesPullingTime="20.645857326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:22.019955163 +0000 UTC m=+82.033057060" watchObservedRunningTime="2026-07-20 09:58:22.019970661 +0000 UTC m=+82.033072557"
Jul 20 09:58:23.001633 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:23.001590    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-8n9sc"
Jul 20 09:58:23.870826 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:23.870722    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c56464c7b-b9pvp"
Jul 20 09:58:23.986848 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:23.986774    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b" event={"ID":"c93d2f2a-5a06-4bea-a170-db5aaa435782","Type":"ContainerStarted","Data":"98b915e8db2c25510ffe25de0701531b149deccf9aa2fa02beacc86656ab7056"}
Jul 20 09:58:24.993776 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:24.993732    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b" event={"ID":"c93d2f2a-5a06-4bea-a170-db5aaa435782","Type":"ContainerStarted","Data":"16e3db222cdeb15d368a7144a880ccf9d315d1793d52f9ccc2ac018d155883e1"}
Jul 20 09:58:25.011875 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:25.011827    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9jx8b" podStartSLOduration=11.066614503 podStartE2EDuration="13.011805234s" totalImagesPullingTime="1.945190731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:25.010892125 +0000 UTC m=+85.023994022" watchObservedRunningTime="2026-07-20 09:58:25.011805234 +0000 UTC m=+85.024907131"
Jul 20 09:58:27.347506 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.347469    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w6m2w"]
Jul 20 09:58:27.422912 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.422883    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.426458 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.426435    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 20 09:58:27.426622 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.426603    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 20 09:58:27.426954 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.426454    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 20 09:58:27.427273 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.427260    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2kv7c\""
Jul 20 09:58:27.571980 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.571932    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-tls\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.572308 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.572285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-root\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.572477 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.572426    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8jc\" (UniqueName: \"kubernetes.io/projected/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-kube-api-access-df8jc\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.572637 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.572607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.572789 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.572772    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-textfile\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.572940 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.572909    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-metrics-client-ca\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.573061 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.573041    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.573175 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.573163    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-wtmp\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.573287 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.573275    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-sys\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.674260 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.674153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-sys\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.674574 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.674519    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-tls\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.674732 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.674716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-root\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.674867 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.674852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8jc\" (UniqueName: \"kubernetes.io/projected/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-kube-api-access-df8jc\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.675012 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.674997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.675148 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.675134    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-textfile\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.675265 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.675251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-metrics-client-ca\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.675394 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.675378    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.675519 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.675506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-wtmp\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.675850 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.675831    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-wtmp\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.676042 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.676003    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-sys\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.676273 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:58:27.676257    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 20 09:58:27.676452 ip-10-0-138-196 kubenswrapper[2572]: E0720 09:58:27.676441    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-tls podName:7ec3ea3d-aff0-4370-b384-5c59e9c08ba6 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:28.176401401 +0000 UTC m=+88.189503282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-tls") pod "node-exporter-w6m2w" (UID: "7ec3ea3d-aff0-4370-b384-5c59e9c08ba6") : secret "node-exporter-tls" not found
Jul 20 09:58:27.676647 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.676631    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-root\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.678106 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.678084    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-metrics-client-ca\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.678271 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.678247    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-textfile\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.678658 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.678635    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.681077 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.680999    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:27.689069 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:27.689028    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8jc\" (UniqueName: \"kubernetes.io/projected/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-kube-api-access-df8jc\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:28.180308 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:28.180274    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-tls\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:28.182968 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:28.182943    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ec3ea3d-aff0-4370-b384-5c59e9c08ba6-node-exporter-tls\") pod \"node-exporter-w6m2w\" (UID: \"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6\") " pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:28.341081 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:28.341036    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w6m2w"
Jul 20 09:58:28.385319 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:28.385286    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec3ea3d_aff0_4370_b384_5c59e9c08ba6.slice/crio-2957714ecf50617eb6df6b164f8759f308ff3410c85eed975b8b35778d228db0 WatchSource:0}: Error finding container 2957714ecf50617eb6df6b164f8759f308ff3410c85eed975b8b35778d228db0: Status 404 returned error can't find the container with id 2957714ecf50617eb6df6b164f8759f308ff3410c85eed975b8b35778d228db0
Jul 20 09:58:29.014273 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:29.014228    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6m2w" event={"ID":"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6","Type":"ContainerStarted","Data":"2957714ecf50617eb6df6b164f8759f308ff3410c85eed975b8b35778d228db0"}
Jul 20 09:58:31.022535 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:31.022497    2572 generic.go:356] "Generic (PLEG): container finished" podID="7ec3ea3d-aff0-4370-b384-5c59e9c08ba6" containerID="b53f12b7bfb8a0396f76bfb369b9d6b50fd6c3778793834fc327624c47ea2a1f" exitCode=0
Jul 20 09:58:31.022919 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:31.022585    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6m2w" event={"ID":"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6","Type":"ContainerDied","Data":"b53f12b7bfb8a0396f76bfb369b9d6b50fd6c3778793834fc327624c47ea2a1f"}
Jul 20 09:58:32.028450 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:32.028411    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6m2w" event={"ID":"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6","Type":"ContainerStarted","Data":"9dc9604b11825e508a1b6c670f82f74f4af6d2a307de16d2ebe61cb6d1e492eb"}
Jul 20 09:58:32.028941 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:32.028505    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6m2w" event={"ID":"7ec3ea3d-aff0-4370-b384-5c59e9c08ba6","Type":"ContainerStarted","Data":"2328407e538ecd0ced3f7ec9698317bb7f84365d029af6cd1a683e97e042d1e4"}
Jul 20 09:58:32.054312 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:32.054267    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w6m2w" podStartSLOduration=3.3730169930000002 podStartE2EDuration="5.054252878s" totalImagesPullingTime="1.569214877s" totalInitContainerRuntime="112.021008ms" isStatefulPod=true podCreationTimestamp="2026-07-20 09:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:32.052953192 +0000 UTC m=+92.066055088" watchObservedRunningTime="2026-07-20 09:58:32.054252878 +0000 UTC m=+92.067354797"
Jul 20 09:58:33.090774 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.090741    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54db66fcff-r8bjw"]
Jul 20 09:58:33.116518 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.116489    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54db66fcff-r8bjw"]
Jul 20 09:58:33.116672 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.116600    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.119640 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.119618    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 20 09:58:33.120675 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.120656    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 20 09:58:33.121015 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.120995    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 20 09:58:33.121015 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121009    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 20 09:58:33.121154 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121073    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 20 09:58:33.121302 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121284    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-oauth-config\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.121341 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121309    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-trusted-ca-bundle\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.121341 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121329    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-config\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.121416 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121350    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-oauth-serving-cert\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.121495 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121482    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sr44\" (UniqueName: \"kubernetes.io/projected/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-kube-api-access-7sr44\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.121608 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-service-ca\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.121794 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-serving-cert\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.121995 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.121976    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4nmtb\""
Jul 20 09:58:33.127532 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.127510    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 20 09:58:33.222620 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.222581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-oauth-config\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.222793 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.222630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-trusted-ca-bundle\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.222793 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.222662    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-config\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.222793 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.222686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-oauth-serving-cert\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.222793 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.222730    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr44\" (UniqueName: \"kubernetes.io/projected/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-kube-api-access-7sr44\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.222793 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.222773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-service-ca\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.223022 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.222797    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-serving-cert\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.223511 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.223468    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-config\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.223511 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.223505    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-oauth-serving-cert\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.223699 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.223598    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-trusted-ca-bundle\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.223699 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.223650    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-service-ca\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.224995 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.224976    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-oauth-config\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.225207 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.225193    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-serving-cert\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.262298 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.262269    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sr44\" (UniqueName: \"kubernetes.io/projected/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-kube-api-access-7sr44\") pod \"console-54db66fcff-r8bjw\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") " pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.426438 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.426345    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:33.608258 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:33.608224    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54db66fcff-r8bjw"]
Jul 20 09:58:33.612468 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:58:33.612442    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa121eb_0d2d_405a_8a04_bbeb7fa7715c.slice/crio-8cd6bd1c7096b1b9d9f0b08cdad1eb2a4120e2693acbe601c4a989d82fce2d9b WatchSource:0}: Error finding container 8cd6bd1c7096b1b9d9f0b08cdad1eb2a4120e2693acbe601c4a989d82fce2d9b: Status 404 returned error can't find the container with id 8cd6bd1c7096b1b9d9f0b08cdad1eb2a4120e2693acbe601c4a989d82fce2d9b
Jul 20 09:58:34.038176 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:34.038146    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54db66fcff-r8bjw" event={"ID":"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c","Type":"ContainerStarted","Data":"8cd6bd1c7096b1b9d9f0b08cdad1eb2a4120e2693acbe601c4a989d82fce2d9b"}
Jul 20 09:58:37.051044 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:37.050958    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54db66fcff-r8bjw" event={"ID":"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c","Type":"ContainerStarted","Data":"c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf"}
Jul 20 09:58:37.073039 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:37.072957    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54db66fcff-r8bjw" podStartSLOduration=0.886340585 podStartE2EDuration="4.072919441s" totalImagesPullingTime="3.186578856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:37.072354943 +0000 UTC m=+97.085456838" watchObservedRunningTime="2026-07-20 09:58:37.072919441 +0000 UTC m=+97.086021339"
Jul 20 09:58:43.426930 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:43.426884    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:43.427511 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:43.427137    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:43.431815 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:43.431793    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:44.080249 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:44.080217    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 09:58:52.982263 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:58:52.982232    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9ztdl"
Jul 20 09:59:56.092949 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.092868    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-678cd75d85-nj2hb"]
Jul 20 09:59:56.098920 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.098894    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.109550 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.109527    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678cd75d85-nj2hb"]
Jul 20 09:59:56.139610 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.139583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-oauth-config\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.139783 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.139614    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-serving-cert\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.139783 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.139641    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-oauth-serving-cert\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.139783 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.139735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-service-ca\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.139783 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.139769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-trusted-ca-bundle\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.139996 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.139813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hr4s\" (UniqueName: \"kubernetes.io/projected/8939a85d-b42c-41d8-88bb-26cca9ade99c-kube-api-access-4hr4s\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.139996 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.139836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-config\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.240491 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.240457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-service-ca\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.240777 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.240756    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-trusted-ca-bundle\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.241632 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.241605    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-trusted-ca-bundle\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.241802 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.241652    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hr4s\" (UniqueName: \"kubernetes.io/projected/8939a85d-b42c-41d8-88bb-26cca9ade99c-kube-api-access-4hr4s\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.241900 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.241821    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-config\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.241900 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.241328    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-service-ca\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.241900 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.241859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-oauth-config\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.241900 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.241887    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-serving-cert\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.242100 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.241924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-oauth-serving-cert\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.242437 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.242415    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-config\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.242529 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.242436    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-oauth-serving-cert\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.244174 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.244144    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-oauth-config\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.244312 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.244293    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-serving-cert\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.252242 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.252221    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hr4s\" (UniqueName: \"kubernetes.io/projected/8939a85d-b42c-41d8-88bb-26cca9ade99c-kube-api-access-4hr4s\") pod \"console-678cd75d85-nj2hb\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") " pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.408182 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.408077    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 09:59:56.556727 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:56.556699    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678cd75d85-nj2hb"]
Jul 20 09:59:56.559399 ip-10-0-138-196 kubenswrapper[2572]: W0720 09:59:56.559372    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8939a85d_b42c_41d8_88bb_26cca9ade99c.slice/crio-aef068f74cbb99591b075fa698134be9d06290bcb6156bbaa84c6276a837c1c0 WatchSource:0}: Error finding container aef068f74cbb99591b075fa698134be9d06290bcb6156bbaa84c6276a837c1c0: Status 404 returned error can't find the container with id aef068f74cbb99591b075fa698134be9d06290bcb6156bbaa84c6276a837c1c0
Jul 20 09:59:57.331885 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:57.331850    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678cd75d85-nj2hb" event={"ID":"8939a85d-b42c-41d8-88bb-26cca9ade99c","Type":"ContainerStarted","Data":"dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e"}
Jul 20 09:59:57.332254 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:57.331900    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678cd75d85-nj2hb" event={"ID":"8939a85d-b42c-41d8-88bb-26cca9ade99c","Type":"ContainerStarted","Data":"aef068f74cbb99591b075fa698134be9d06290bcb6156bbaa84c6276a837c1c0"}
Jul 20 09:59:57.352364 ip-10-0-138-196 kubenswrapper[2572]: I0720 09:59:57.352318    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-678cd75d85-nj2hb" podStartSLOduration=1.35230219 podStartE2EDuration="1.35230219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:57.351524181 +0000 UTC m=+177.364626101" watchObservedRunningTime="2026-07-20 09:59:57.35230219 +0000 UTC m=+177.365404086"
Jul 20 10:00:06.409030 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:06.408986    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 10:00:06.409477 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:06.409083    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 10:00:06.413958 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:06.413932    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 10:00:07.369423 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:07.369396    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 10:00:07.420706 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:07.420678    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54db66fcff-r8bjw"]
Jul 20 10:00:32.442033 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.441962    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54db66fcff-r8bjw" podUID="dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" containerName="console" containerID="cri-o://c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf" gracePeriod=15
Jul 20 10:00:32.683176 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.683153    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 10:00:32.810730 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.810699    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-oauth-serving-cert\") pod \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") "
Jul 20 10:00:32.810730 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.810734    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-service-ca\") pod \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") "
Jul 20 10:00:32.810959 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.810764    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-trusted-ca-bundle\") pod \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") "
Jul 20 10:00:32.810959 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.810782    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-config\") pod \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") "
Jul 20 10:00:32.810959 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.810803    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-oauth-config\") pod \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") "
Jul 20 10:00:32.810959 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.810846    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sr44\" (UniqueName: \"kubernetes.io/projected/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-kube-api-access-7sr44\") pod \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") "
Jul 20 10:00:32.810959 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.810871    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-serving-cert\") pod \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\" (UID: \"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c\") "
Jul 20 10:00:32.811350 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.811266    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-config" (OuterVolumeSpecName: "console-config") pod "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" (UID: "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:32.811350 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.811287    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" (UID: "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:32.811350 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.811303    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-service-ca" (OuterVolumeSpecName: "service-ca") pod "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" (UID: "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:32.811539 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.811421    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" (UID: "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:32.813215 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.813176    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-kube-api-access-7sr44" (OuterVolumeSpecName: "kube-api-access-7sr44") pod "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" (UID: "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c"). InnerVolumeSpecName "kube-api-access-7sr44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:00:32.813215 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.813206    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" (UID: "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:32.813366 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.813226    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" (UID: "dfa121eb-0d2d-405a-8a04-bbeb7fa7715c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:32.912134 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.912094    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sr44\" (UniqueName: \"kubernetes.io/projected/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-kube-api-access-7sr44\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:00:32.912134 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.912128    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-serving-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:00:32.912134 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.912137    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-oauth-serving-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:00:32.912134 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.912146    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-service-ca\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:00:32.912440 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.912155    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-trusted-ca-bundle\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:00:32.912440 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.912163    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-config\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:00:32.912440 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:32.912171    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c-console-oauth-config\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:00:33.455271 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.455236    2572 generic.go:356] "Generic (PLEG): container finished" podID="dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" containerID="c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf" exitCode=0
Jul 20 10:00:33.455681 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.455276    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54db66fcff-r8bjw" event={"ID":"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c","Type":"ContainerDied","Data":"c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf"}
Jul 20 10:00:33.455681 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.455283    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54db66fcff-r8bjw"
Jul 20 10:00:33.455681 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.455327    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54db66fcff-r8bjw" event={"ID":"dfa121eb-0d2d-405a-8a04-bbeb7fa7715c","Type":"ContainerDied","Data":"8cd6bd1c7096b1b9d9f0b08cdad1eb2a4120e2693acbe601c4a989d82fce2d9b"}
Jul 20 10:00:33.456199 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.456184    2572 scope.go:117] "RemoveContainer" containerID="c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf"
Jul 20 10:00:33.464758 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.464737    2572 scope.go:117] "RemoveContainer" containerID="c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf"
Jul 20 10:00:33.465032 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:00:33.465014    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf\": container with ID starting with c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf not found: ID does not exist" containerID="c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf"
Jul 20 10:00:33.465084 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.465041    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf"} err="failed to get container status \"c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf\": rpc error: code = NotFound desc = could not find container \"c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf\": container with ID starting with c4d2619a756fcf95e8ecfc02bf90acf2e30b7090b60b08a1b8ee5dbc84ce9ddf not found: ID does not exist"
Jul 20 10:00:33.475803 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.475779    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54db66fcff-r8bjw"]
Jul 20 10:00:33.479445 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:33.479423    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54db66fcff-r8bjw"]
Jul 20 10:00:34.519510 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:00:34.519476    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" path="/var/lib/kubelet/pods/dfa121eb-0d2d-405a-8a04-bbeb7fa7715c/volumes"
Jul 20 10:01:06.697282 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.697243    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-99c8876f6-v6j45"]
Jul 20 10:01:06.697784 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.697524    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" containerName="console"
Jul 20 10:01:06.697784 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.697535    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" containerName="console"
Jul 20 10:01:06.697784 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.697597    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa121eb-0d2d-405a-8a04-bbeb7fa7715c" containerName="console"
Jul 20 10:01:06.700124 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.700107    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.713063 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.713035    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99c8876f6-v6j45"]
Jul 20 10:01:06.860532 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.860495    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-oauth-config\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.860742 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.860541    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-oauth-serving-cert\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.860742 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.860640    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-config\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.860742 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.860681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-trusted-ca-bundle\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.860742 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.860701    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-serving-cert\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.860891 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.860743    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-service-ca\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.860891 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.860767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmm2n\" (UniqueName: \"kubernetes.io/projected/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-kube-api-access-hmm2n\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.961425 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.961385    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmm2n\" (UniqueName: \"kubernetes.io/projected/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-kube-api-access-hmm2n\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.961613 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.961438    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-oauth-config\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.961613 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.961471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-oauth-serving-cert\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.961613 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.961503    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-config\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.961774 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.961606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-trusted-ca-bundle\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.961774 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.961651    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-serving-cert\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.961774 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.961680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-service-ca\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.962256 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.962228    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-config\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.962370 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.962308    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-oauth-serving-cert\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.962370 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.962308    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-service-ca\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.962501 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.962477    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-trusted-ca-bundle\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.964000 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.963978    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-oauth-config\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.964177 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.964161    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-serving-cert\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:06.969283 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:06.969260    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmm2n\" (UniqueName: \"kubernetes.io/projected/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-kube-api-access-hmm2n\") pod \"console-99c8876f6-v6j45\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") " pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:07.009260 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:07.009224    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:07.147733 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:07.147692    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99c8876f6-v6j45"]
Jul 20 10:01:07.150501 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:01:07.150469    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d82127_fe2d_4e9f_b03e_6f95c6ba4bd6.slice/crio-a600c3b9cb2142a9e76ec8eeaebef65b3b4be307e77319f1ae857f04e337115f WatchSource:0}: Error finding container a600c3b9cb2142a9e76ec8eeaebef65b3b4be307e77319f1ae857f04e337115f: Status 404 returned error can't find the container with id a600c3b9cb2142a9e76ec8eeaebef65b3b4be307e77319f1ae857f04e337115f
Jul 20 10:01:07.570902 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:07.570870    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99c8876f6-v6j45" event={"ID":"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6","Type":"ContainerStarted","Data":"5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447"}
Jul 20 10:01:07.571072 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:07.570916    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99c8876f6-v6j45" event={"ID":"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6","Type":"ContainerStarted","Data":"a600c3b9cb2142a9e76ec8eeaebef65b3b4be307e77319f1ae857f04e337115f"}
Jul 20 10:01:07.592512 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:07.592467    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-99c8876f6-v6j45" podStartSLOduration=1.592449558 podStartE2EDuration="1.592449558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:01:07.592259871 +0000 UTC m=+247.605361757" watchObservedRunningTime="2026-07-20 10:01:07.592449558 +0000 UTC m=+247.605551453"
Jul 20 10:01:17.010207 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:17.010155    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:17.010720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:17.010262    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:17.014769 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:17.014746    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:17.611273 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:17.611248    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:01:17.667369 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:17.667340    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678cd75d85-nj2hb"]
Jul 20 10:01:42.687806 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:42.687762    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-678cd75d85-nj2hb" podUID="8939a85d-b42c-41d8-88bb-26cca9ade99c" containerName="console" containerID="cri-o://dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e" gracePeriod=15
Jul 20 10:01:42.928796 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:42.928772    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 10:01:43.023579 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.023526    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-oauth-serving-cert\") pod \"8939a85d-b42c-41d8-88bb-26cca9ade99c\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") "
Jul 20 10:01:43.023579 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.023582    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-oauth-config\") pod \"8939a85d-b42c-41d8-88bb-26cca9ade99c\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") "
Jul 20 10:01:43.023812 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.023607    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-service-ca\") pod \"8939a85d-b42c-41d8-88bb-26cca9ade99c\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") "
Jul 20 10:01:43.023812 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.023646    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-serving-cert\") pod \"8939a85d-b42c-41d8-88bb-26cca9ade99c\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") "
Jul 20 10:01:43.023812 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.023667    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-config\") pod \"8939a85d-b42c-41d8-88bb-26cca9ade99c\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") "
Jul 20 10:01:43.023812 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.023682    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-trusted-ca-bundle\") pod \"8939a85d-b42c-41d8-88bb-26cca9ade99c\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") "
Jul 20 10:01:43.023812 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.023704    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hr4s\" (UniqueName: \"kubernetes.io/projected/8939a85d-b42c-41d8-88bb-26cca9ade99c-kube-api-access-4hr4s\") pod \"8939a85d-b42c-41d8-88bb-26cca9ade99c\" (UID: \"8939a85d-b42c-41d8-88bb-26cca9ade99c\") "
Jul 20 10:01:43.024106 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.023951    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8939a85d-b42c-41d8-88bb-26cca9ade99c" (UID: "8939a85d-b42c-41d8-88bb-26cca9ade99c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:01:43.024206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.024142    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-service-ca" (OuterVolumeSpecName: "service-ca") pod "8939a85d-b42c-41d8-88bb-26cca9ade99c" (UID: "8939a85d-b42c-41d8-88bb-26cca9ade99c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:01:43.024206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.024151    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-config" (OuterVolumeSpecName: "console-config") pod "8939a85d-b42c-41d8-88bb-26cca9ade99c" (UID: "8939a85d-b42c-41d8-88bb-26cca9ade99c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:01:43.024206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.024171    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8939a85d-b42c-41d8-88bb-26cca9ade99c" (UID: "8939a85d-b42c-41d8-88bb-26cca9ade99c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:01:43.025819 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.025797    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8939a85d-b42c-41d8-88bb-26cca9ade99c" (UID: "8939a85d-b42c-41d8-88bb-26cca9ade99c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:01:43.025936 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.025898    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8939a85d-b42c-41d8-88bb-26cca9ade99c" (UID: "8939a85d-b42c-41d8-88bb-26cca9ade99c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:01:43.026032 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.025980    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8939a85d-b42c-41d8-88bb-26cca9ade99c-kube-api-access-4hr4s" (OuterVolumeSpecName: "kube-api-access-4hr4s") pod "8939a85d-b42c-41d8-88bb-26cca9ade99c" (UID: "8939a85d-b42c-41d8-88bb-26cca9ade99c"). InnerVolumeSpecName "kube-api-access-4hr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:01:43.125091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.125055    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-serving-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:01:43.125091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.125084    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-config\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:01:43.125091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.125098    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-trusted-ca-bundle\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:01:43.125317 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.125112    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hr4s\" (UniqueName: \"kubernetes.io/projected/8939a85d-b42c-41d8-88bb-26cca9ade99c-kube-api-access-4hr4s\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:01:43.125317 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.125123    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-oauth-serving-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:01:43.125317 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.125136    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8939a85d-b42c-41d8-88bb-26cca9ade99c-console-oauth-config\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:01:43.125317 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.125147    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8939a85d-b42c-41d8-88bb-26cca9ade99c-service-ca\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:01:43.696719 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.696683    2572 generic.go:356] "Generic (PLEG): container finished" podID="8939a85d-b42c-41d8-88bb-26cca9ade99c" containerID="dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e" exitCode=0
Jul 20 10:01:43.696719 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.696723    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678cd75d85-nj2hb"
Jul 20 10:01:43.697201 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.696747    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678cd75d85-nj2hb" event={"ID":"8939a85d-b42c-41d8-88bb-26cca9ade99c","Type":"ContainerDied","Data":"dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e"}
Jul 20 10:01:43.697201 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.696795    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678cd75d85-nj2hb" event={"ID":"8939a85d-b42c-41d8-88bb-26cca9ade99c","Type":"ContainerDied","Data":"aef068f74cbb99591b075fa698134be9d06290bcb6156bbaa84c6276a837c1c0"}
Jul 20 10:01:43.697201 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.696825    2572 scope.go:117] "RemoveContainer" containerID="dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e"
Jul 20 10:01:43.704977 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.704961    2572 scope.go:117] "RemoveContainer" containerID="dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e"
Jul 20 10:01:43.705250 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:01:43.705232    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e\": container with ID starting with dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e not found: ID does not exist" containerID="dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e"
Jul 20 10:01:43.705303 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.705258    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e"} err="failed to get container status \"dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e\": rpc error: code = NotFound desc = could not find container \"dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e\": container with ID starting with dc364f81de7af45458a83bad1d703f053ccf801130ded08371c2a9cba458796e not found: ID does not exist"
Jul 20 10:01:43.716879 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.716851    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678cd75d85-nj2hb"]
Jul 20 10:01:43.718491 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:43.718471    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-678cd75d85-nj2hb"]
Jul 20 10:01:44.518884 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:01:44.518855    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8939a85d-b42c-41d8-88bb-26cca9ade99c" path="/var/lib/kubelet/pods/8939a85d-b42c-41d8-88bb-26cca9ade99c/volumes"
Jul 20 10:02:00.369266 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:02:00.369235    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:02:00.369768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:02:00.369748    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:02:00.372961 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:02:00.372943    2572 kubelet.go:1648] "Image garbage collection succeeded"
Jul 20 10:03:20.966318 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.966285    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"]
Jul 20 10:03:20.966799 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.966546    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8939a85d-b42c-41d8-88bb-26cca9ade99c" containerName="console"
Jul 20 10:03:20.966799 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.966568    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8939a85d-b42c-41d8-88bb-26cca9ade99c" containerName="console"
Jul 20 10:03:20.966799 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.966610    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8939a85d-b42c-41d8-88bb-26cca9ade99c" containerName="console"
Jul 20 10:03:20.968296 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.968280    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:20.970954 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.970930    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 20 10:03:20.971084 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.971025    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 20 10:03:20.972217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.972194    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-m8g5k\""
Jul 20 10:03:20.972338 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.972226    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 20 10:03:20.972338 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.972234    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 20 10:03:20.972441 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.972428    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 20 10:03:20.980730 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:20.980706    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"]
Jul 20 10:03:21.064223 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.064179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3f742676-af59-4a1b-af85-8e152cade7b6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.064396 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.064244    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pcv\" (UniqueName: \"kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-kube-api-access-c7pcv\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.064396 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.064290    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-certificates\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.165123 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.165091    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3f742676-af59-4a1b-af85-8e152cade7b6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.165294 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.165136    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pcv\" (UniqueName: \"kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-kube-api-access-c7pcv\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.165294 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.165199    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-certificates\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.165369 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:03:21.165298    2572 secret.go:281] references non-existent secret key: tls.crt
Jul 20 10:03:21.165369 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:03:21.165311    2572 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 20 10:03:21.165369 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:03:21.165326    2572 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-wt86z: references non-existent secret key: tls.crt
Jul 20 10:03:21.165369 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:03:21.165368    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-certificates podName:3f742676-af59-4a1b-af85-8e152cade7b6 nodeName:}" failed. No retries permitted until 2026-07-20 10:03:21.66535476 +0000 UTC m=+381.678456634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-certificates") pod "keda-metrics-apiserver-55557df8c-wt86z" (UID: "3f742676-af59-4a1b-af85-8e152cade7b6") : references non-existent secret key: tls.crt
Jul 20 10:03:21.165709 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.165686    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3f742676-af59-4a1b-af85-8e152cade7b6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.188181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.188150    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pcv\" (UniqueName: \"kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-kube-api-access-c7pcv\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.670263 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.670227    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-certificates\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.673359 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.673325    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3f742676-af59-4a1b-af85-8e152cade7b6-certificates\") pod \"keda-metrics-apiserver-55557df8c-wt86z\" (UID: \"3f742676-af59-4a1b-af85-8e152cade7b6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:21.878460 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:21.878419    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:22.020140 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:22.020106    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"]
Jul 20 10:03:22.023212 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:03:22.023183    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f742676_af59_4a1b_af85_8e152cade7b6.slice/crio-26aa2df7a917eda260d6577ed6778719794d25828e593842f10ae654056c6076 WatchSource:0}: Error finding container 26aa2df7a917eda260d6577ed6778719794d25828e593842f10ae654056c6076: Status 404 returned error can't find the container with id 26aa2df7a917eda260d6577ed6778719794d25828e593842f10ae654056c6076
Jul 20 10:03:22.024847 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:22.024827    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:03:22.032815 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:22.032789    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z" event={"ID":"3f742676-af59-4a1b-af85-8e152cade7b6","Type":"ContainerStarted","Data":"26aa2df7a917eda260d6577ed6778719794d25828e593842f10ae654056c6076"}
Jul 20 10:03:25.045961 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:25.045934    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z" event={"ID":"3f742676-af59-4a1b-af85-8e152cade7b6","Type":"ContainerStarted","Data":"985912e35317dc787ebf4e17124c480371ffb9f391891e42ba6fab7e9cf3edc2"}
Jul 20 10:03:25.046370 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:25.046041    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:03:25.063843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:25.063799    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z" podStartSLOduration=2.118004881 podStartE2EDuration="5.063782354s" totalImagesPullingTime="2.945777473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:03:25.062219364 +0000 UTC m=+385.075321262" watchObservedRunningTime="2026-07-20 10:03:25.063782354 +0000 UTC m=+385.076884250"
Jul 20 10:03:36.054733 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:03:36.054702    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wt86z"
Jul 20 10:04:36.369437 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.369356    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-l9pg7"]
Jul 20 10:04:36.371914 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.371897    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:36.375148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.375123    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 20 10:04:36.375533 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.375514    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 20 10:04:36.375698 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.375680    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 20 10:04:36.375864 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.375842    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-jgfcs\""
Jul 20 10:04:36.397201 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.397173    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-l9pg7"]
Jul 20 10:04:36.433278 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.433251    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-cert\") pod \"kserve-controller-manager-645f4f9567-l9pg7\" (UID: \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\") " pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:36.433407 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.433298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlxv\" (UniqueName: \"kubernetes.io/projected/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-kube-api-access-bzlxv\") pod \"kserve-controller-manager-645f4f9567-l9pg7\" (UID: \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\") " pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:36.458104 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.458076    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-5d54f9649b-7wc28"]
Jul 20 10:04:36.460756 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.460741    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:36.469904 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.469885    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 20 10:04:36.471836 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.471820    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-nwbd5\""
Jul 20 10:04:36.481405 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.481381    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5d54f9649b-7wc28"]
Jul 20 10:04:36.533894 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.533851    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrws\" (UniqueName: \"kubernetes.io/projected/145569bb-e0da-4065-886a-c3a8a5bcb6b4-kube-api-access-4qrws\") pod \"llmisvc-controller-manager-5d54f9649b-7wc28\" (UID: \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\") " pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:36.533894 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.533898    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145569bb-e0da-4065-886a-c3a8a5bcb6b4-cert\") pod \"llmisvc-controller-manager-5d54f9649b-7wc28\" (UID: \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\") " pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:36.534107 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.533934    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-cert\") pod \"kserve-controller-manager-645f4f9567-l9pg7\" (UID: \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\") " pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:36.534107 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.533974    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlxv\" (UniqueName: \"kubernetes.io/projected/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-kube-api-access-bzlxv\") pod \"kserve-controller-manager-645f4f9567-l9pg7\" (UID: \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\") " pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:36.536351 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.536332    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-cert\") pod \"kserve-controller-manager-645f4f9567-l9pg7\" (UID: \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\") " pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:36.571067 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.571042    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlxv\" (UniqueName: \"kubernetes.io/projected/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-kube-api-access-bzlxv\") pod \"kserve-controller-manager-645f4f9567-l9pg7\" (UID: \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\") " pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:36.635318 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.635243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrws\" (UniqueName: \"kubernetes.io/projected/145569bb-e0da-4065-886a-c3a8a5bcb6b4-kube-api-access-4qrws\") pod \"llmisvc-controller-manager-5d54f9649b-7wc28\" (UID: \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\") " pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:36.635318 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.635276    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145569bb-e0da-4065-886a-c3a8a5bcb6b4-cert\") pod \"llmisvc-controller-manager-5d54f9649b-7wc28\" (UID: \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\") " pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:36.637692 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.637669    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145569bb-e0da-4065-886a-c3a8a5bcb6b4-cert\") pod \"llmisvc-controller-manager-5d54f9649b-7wc28\" (UID: \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\") " pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:36.651787 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.651762    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrws\" (UniqueName: \"kubernetes.io/projected/145569bb-e0da-4065-886a-c3a8a5bcb6b4-kube-api-access-4qrws\") pod \"llmisvc-controller-manager-5d54f9649b-7wc28\" (UID: \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\") " pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:36.681863 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.681833    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:36.771876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.771844    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:36.827123 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.827083    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-l9pg7"]
Jul 20 10:04:36.829924 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:04:36.829894    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83ecc8e_fa95_4b33_aa32_241c18b4e22e.slice/crio-8f20a5c3fa0348e81985dfcecf37aeca52c613e386048832e15d3318b5651015 WatchSource:0}: Error finding container 8f20a5c3fa0348e81985dfcecf37aeca52c613e386048832e15d3318b5651015: Status 404 returned error can't find the container with id 8f20a5c3fa0348e81985dfcecf37aeca52c613e386048832e15d3318b5651015
Jul 20 10:04:36.906368 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:36.906284    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5d54f9649b-7wc28"]
Jul 20 10:04:36.909275 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:04:36.909251    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod145569bb_e0da_4065_886a_c3a8a5bcb6b4.slice/crio-e4311ed3785e958960eab680f932d6e43ede7b3f62b806a949bb4636fde121b6 WatchSource:0}: Error finding container e4311ed3785e958960eab680f932d6e43ede7b3f62b806a949bb4636fde121b6: Status 404 returned error can't find the container with id e4311ed3785e958960eab680f932d6e43ede7b3f62b806a949bb4636fde121b6
Jul 20 10:04:37.299114 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:37.299073    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7" event={"ID":"c83ecc8e-fa95-4b33-aa32-241c18b4e22e","Type":"ContainerStarted","Data":"8f20a5c3fa0348e81985dfcecf37aeca52c613e386048832e15d3318b5651015"}
Jul 20 10:04:37.300133 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:37.300114    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28" event={"ID":"145569bb-e0da-4065-886a-c3a8a5bcb6b4","Type":"ContainerStarted","Data":"e4311ed3785e958960eab680f932d6e43ede7b3f62b806a949bb4636fde121b6"}
Jul 20 10:04:41.318256 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:41.318216    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7" event={"ID":"c83ecc8e-fa95-4b33-aa32-241c18b4e22e","Type":"ContainerStarted","Data":"07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d"}
Jul 20 10:04:41.318718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:41.318314    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:04:41.319771 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:41.319747    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28" event={"ID":"145569bb-e0da-4065-886a-c3a8a5bcb6b4","Type":"ContainerStarted","Data":"9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f"}
Jul 20 10:04:41.319869 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:41.319794    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:04:41.338035 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:41.337993    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7" podStartSLOduration=1.73257268 podStartE2EDuration="5.337978575s" totalImagesPullingTime="3.605405895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:04:41.336194745 +0000 UTC m=+461.349296642" watchObservedRunningTime="2026-07-20 10:04:41.337978575 +0000 UTC m=+461.351080474"
Jul 20 10:04:41.361639 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:04:41.361597    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28" podStartSLOduration=1.8366372960000001 podStartE2EDuration="5.361583929s" totalImagesPullingTime="3.524946633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:04:41.359675058 +0000 UTC m=+461.372776952" watchObservedRunningTime="2026-07-20 10:04:41.361583929 +0000 UTC m=+461.374685825"
Jul 20 10:05:10.068831 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:10.068799    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-99c8876f6-v6j45"]
Jul 20 10:05:12.326294 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:12.326249    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:05:12.330324 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:12.330294    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:05:35.095104 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.095044    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-99c8876f6-v6j45" podUID="75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" containerName="console" containerID="cri-o://5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447" gracePeriod=15
Jul 20 10:05:35.338489 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.338466    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:05:35.409110 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409028    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-config\") pod \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") "
Jul 20 10:05:35.409110 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409073    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-oauth-config\") pod \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") "
Jul 20 10:05:35.409110 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409091    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-service-ca\") pod \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") "
Jul 20 10:05:35.409389 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409125    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-oauth-serving-cert\") pod \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") "
Jul 20 10:05:35.409389 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409147    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmm2n\" (UniqueName: \"kubernetes.io/projected/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-kube-api-access-hmm2n\") pod \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") "
Jul 20 10:05:35.409389 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409175    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-serving-cert\") pod \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") "
Jul 20 10:05:35.409389 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409227    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-trusted-ca-bundle\") pod \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\" (UID: \"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6\") "
Jul 20 10:05:35.409619 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409580    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-service-ca" (OuterVolumeSpecName: "service-ca") pod "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" (UID: "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:05:35.409619 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409598    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-config" (OuterVolumeSpecName: "console-config") pod "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" (UID: "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:05:35.409701 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409629    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" (UID: "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:05:35.409739 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.409698    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" (UID: "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:05:35.411254 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.411233    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" (UID: "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:05:35.411321 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.411266    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" (UID: "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:05:35.411356 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.411323    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-kube-api-access-hmm2n" (OuterVolumeSpecName: "kube-api-access-hmm2n") pod "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" (UID: "75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6"). InnerVolumeSpecName "kube-api-access-hmm2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:05:35.509827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.509791    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-config\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:05:35.509827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.509822    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-oauth-config\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:05:35.509827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.509832    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-service-ca\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:05:35.509827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.509840    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-oauth-serving-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:05:35.510075 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.509848    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmm2n\" (UniqueName: \"kubernetes.io/projected/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-kube-api-access-hmm2n\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:05:35.510075 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.509858    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-console-serving-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:05:35.510075 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.509867    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6-trusted-ca-bundle\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:05:35.516972 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.516942    2572 generic.go:356] "Generic (PLEG): container finished" podID="75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" containerID="5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447" exitCode=0
Jul 20 10:05:35.517093 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.516981    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99c8876f6-v6j45"
Jul 20 10:05:35.517093 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.516994    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99c8876f6-v6j45" event={"ID":"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6","Type":"ContainerDied","Data":"5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447"}
Jul 20 10:05:35.517093 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.517031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99c8876f6-v6j45" event={"ID":"75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6","Type":"ContainerDied","Data":"a600c3b9cb2142a9e76ec8eeaebef65b3b4be307e77319f1ae857f04e337115f"}
Jul 20 10:05:35.517093 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.517057    2572 scope.go:117] "RemoveContainer" containerID="5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447"
Jul 20 10:05:35.525309 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.525293    2572 scope.go:117] "RemoveContainer" containerID="5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447"
Jul 20 10:05:35.525579 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:05:35.525541    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447\": container with ID starting with 5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447 not found: ID does not exist" containerID="5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447"
Jul 20 10:05:35.525652 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.525594    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447"} err="failed to get container status \"5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447\": rpc error: code = NotFound desc = could not find container \"5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447\": container with ID starting with 5aa166a519c017f20687bc6a70b3c43284b9212e20179cbd3bcb996b5654b447 not found: ID does not exist"
Jul 20 10:05:35.565569 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.565524    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-99c8876f6-v6j45"]
Jul 20 10:05:35.583749 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:35.583724    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-99c8876f6-v6j45"]
Jul 20 10:05:36.519445 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:36.519414    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" path="/var/lib/kubelet/pods/75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6/volumes"
Jul 20 10:05:48.695177 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.695139    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-msj2s"]
Jul 20 10:05:48.695658 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.695430    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" containerName="console"
Jul 20 10:05:48.695658 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.695446    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" containerName="console"
Jul 20 10:05:48.695658 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.695500    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d82127-fe2d-4e9f-b03e-6f95c6ba4bd6" containerName="console"
Jul 20 10:05:48.698160 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.698139    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-msj2s"
Jul 20 10:05:48.701978 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.701952    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 20 10:05:48.702792 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.702775    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 20 10:05:48.702854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.702775    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-s27l4\""
Jul 20 10:05:48.704089 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.704054    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168bccb0-6a90-45a3-bc7c-f626f4ec5502-bound-sa-token\") pod \"cert-manager-55b4d954d5-msj2s\" (UID: \"168bccb0-6a90-45a3-bc7c-f626f4ec5502\") " pod="cert-manager/cert-manager-55b4d954d5-msj2s"
Jul 20 10:05:48.704196 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.704142    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxfbr\" (UniqueName: \"kubernetes.io/projected/168bccb0-6a90-45a3-bc7c-f626f4ec5502-kube-api-access-kxfbr\") pod \"cert-manager-55b4d954d5-msj2s\" (UID: \"168bccb0-6a90-45a3-bc7c-f626f4ec5502\") " pod="cert-manager/cert-manager-55b4d954d5-msj2s"
Jul 20 10:05:48.721366 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.721338    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-msj2s"]
Jul 20 10:05:48.804642 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.804608    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168bccb0-6a90-45a3-bc7c-f626f4ec5502-bound-sa-token\") pod \"cert-manager-55b4d954d5-msj2s\" (UID: \"168bccb0-6a90-45a3-bc7c-f626f4ec5502\") " pod="cert-manager/cert-manager-55b4d954d5-msj2s"
Jul 20 10:05:48.804642 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.804648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxfbr\" (UniqueName: \"kubernetes.io/projected/168bccb0-6a90-45a3-bc7c-f626f4ec5502-kube-api-access-kxfbr\") pod \"cert-manager-55b4d954d5-msj2s\" (UID: \"168bccb0-6a90-45a3-bc7c-f626f4ec5502\") " pod="cert-manager/cert-manager-55b4d954d5-msj2s"
Jul 20 10:05:48.821741 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.821717    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168bccb0-6a90-45a3-bc7c-f626f4ec5502-bound-sa-token\") pod \"cert-manager-55b4d954d5-msj2s\" (UID: \"168bccb0-6a90-45a3-bc7c-f626f4ec5502\") " pod="cert-manager/cert-manager-55b4d954d5-msj2s"
Jul 20 10:05:48.821866 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:48.821777    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxfbr\" (UniqueName: \"kubernetes.io/projected/168bccb0-6a90-45a3-bc7c-f626f4ec5502-kube-api-access-kxfbr\") pod \"cert-manager-55b4d954d5-msj2s\" (UID: \"168bccb0-6a90-45a3-bc7c-f626f4ec5502\") " pod="cert-manager/cert-manager-55b4d954d5-msj2s"
Jul 20 10:05:49.006768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:49.006732    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-msj2s"
Jul 20 10:05:49.149209 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:49.149167    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-msj2s"]
Jul 20 10:05:49.152686 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:05:49.152656    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod168bccb0_6a90_45a3_bc7c_f626f4ec5502.slice/crio-9bc6c09285593177288053bc34f854a355c4d3a4835a3a2e6ae28a4955357f62 WatchSource:0}: Error finding container 9bc6c09285593177288053bc34f854a355c4d3a4835a3a2e6ae28a4955357f62: Status 404 returned error can't find the container with id 9bc6c09285593177288053bc34f854a355c4d3a4835a3a2e6ae28a4955357f62
Jul 20 10:05:49.566255 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:49.566221    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-msj2s" event={"ID":"168bccb0-6a90-45a3-bc7c-f626f4ec5502","Type":"ContainerStarted","Data":"9bc6c09285593177288053bc34f854a355c4d3a4835a3a2e6ae28a4955357f62"}
Jul 20 10:05:52.581064 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:52.581028    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-msj2s" event={"ID":"168bccb0-6a90-45a3-bc7c-f626f4ec5502","Type":"ContainerStarted","Data":"4218de0a6707f824a40d7b8a60175a8f814e45e3c4b8c7e053225c759b8d4bfc"}
Jul 20 10:05:52.615990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:05:52.615948    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-msj2s" podStartSLOduration=1.722714101 podStartE2EDuration="4.61593338s" totalImagesPullingTime="2.893219279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:05:52.614406637 +0000 UTC m=+532.627508533" watchObservedRunningTime="2026-07-20 10:05:52.61593338 +0000 UTC m=+532.629035275"
Jul 20 10:06:27.946857 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.946817    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"]
Jul 20 10:06:27.949123 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.949102    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:27.951875 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.951850    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 20 10:06:27.953049 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.953023    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 20 10:06:27.953049 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.953031    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 20 10:06:27.953212 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.953148    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j7n92\""
Jul 20 10:06:27.953212 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.953160    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 20 10:06:27.953412 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.953392    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 20 10:06:27.961144 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.961123    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"]
Jul 20 10:06:27.994575 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.994533    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36b3e193-fea9-4ce0-ae12-fe593edc284b-metrics-cert\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:27.994681 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.994596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6ct\" (UniqueName: \"kubernetes.io/projected/36b3e193-fea9-4ce0-ae12-fe593edc284b-kube-api-access-4z6ct\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:27.994725 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.994680    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36b3e193-fea9-4ce0-ae12-fe593edc284b-cert\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:27.994725 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:27.994712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/36b3e193-fea9-4ce0-ae12-fe593edc284b-manager-config\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.095848 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.095811    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36b3e193-fea9-4ce0-ae12-fe593edc284b-cert\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.095848 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.095844    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/36b3e193-fea9-4ce0-ae12-fe593edc284b-manager-config\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.096044 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.095873    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36b3e193-fea9-4ce0-ae12-fe593edc284b-metrics-cert\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.096044 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.095909    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6ct\" (UniqueName: \"kubernetes.io/projected/36b3e193-fea9-4ce0-ae12-fe593edc284b-kube-api-access-4z6ct\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.096538 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.096520    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/36b3e193-fea9-4ce0-ae12-fe593edc284b-manager-config\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.098213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.098181    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36b3e193-fea9-4ce0-ae12-fe593edc284b-metrics-cert\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.098315 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.098230    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36b3e193-fea9-4ce0-ae12-fe593edc284b-cert\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.107152 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.107124    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6ct\" (UniqueName: \"kubernetes.io/projected/36b3e193-fea9-4ce0-ae12-fe593edc284b-kube-api-access-4z6ct\") pod \"lws-controller-manager-86dd6ccfdd-9mv22\" (UID: \"36b3e193-fea9-4ce0-ae12-fe593edc284b\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.258657 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.258616    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:28.383593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.383569    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"]
Jul 20 10:06:28.386066 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:06:28.386040    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b3e193_fea9_4ce0_ae12_fe593edc284b.slice/crio-304ebc98d5b5bb81539f5a3d116dd0d88c48c0e4901380f26d32c5403720b180 WatchSource:0}: Error finding container 304ebc98d5b5bb81539f5a3d116dd0d88c48c0e4901380f26d32c5403720b180: Status 404 returned error can't find the container with id 304ebc98d5b5bb81539f5a3d116dd0d88c48c0e4901380f26d32c5403720b180
Jul 20 10:06:28.713573 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:28.713528    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22" event={"ID":"36b3e193-fea9-4ce0-ae12-fe593edc284b","Type":"ContainerStarted","Data":"304ebc98d5b5bb81539f5a3d116dd0d88c48c0e4901380f26d32c5403720b180"}
Jul 20 10:06:30.726765 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:30.726725    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22" event={"ID":"36b3e193-fea9-4ce0-ae12-fe593edc284b","Type":"ContainerStarted","Data":"09878615dd5c8cc556f61eac8afa9b8e9cb4d39c91d2049c49770ae632513613"}
Jul 20 10:06:30.727191 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:30.726986    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:06:30.744644 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:30.744599    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22" podStartSLOduration=1.4758906729999999 podStartE2EDuration="3.74458205s" totalImagesPullingTime="2.268691377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:30.743997869 +0000 UTC m=+570.757099766" watchObservedRunningTime="2026-07-20 10:06:30.74458205 +0000 UTC m=+570.757683946"
Jul 20 10:06:41.732766 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:06:41.732736    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-9mv22"
Jul 20 10:07:00.397247 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:00.397220    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:07:00.398571 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:00.398540    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:07:13.918269 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.918235    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"]
Jul 20 10:07:13.925135 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.925110    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:13.927905 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.927877    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 20 10:07:13.928099 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.928076    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 20 10:07:13.928171 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.928126    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 20 10:07:13.928901 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.928879    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 20 10:07:13.929038 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.928968    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-krh9b\""
Jul 20 10:07:13.929371 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.929350    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"]
Jul 20 10:07:13.955963 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.955922    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:13.956144 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.955971    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pbw\" (UniqueName: \"kubernetes.io/projected/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-kube-api-access-97pbw\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:13.956144 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:13.955998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.057042 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.057000    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.057252 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.057049    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pbw\" (UniqueName: \"kubernetes.io/projected/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-kube-api-access-97pbw\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.057252 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.057077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.057252 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:07:14.057176    2572 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 20 10:07:14.057420 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:07:14.057265    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-plugin-serving-cert podName:4820f7c1-e06a-403e-bbd4-d3b9d3ef0660 nodeName:}" failed. No retries permitted until 2026-07-20 10:07:14.557245209 +0000 UTC m=+614.570347085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-7fxgf" (UID: "4820f7c1-e06a-403e-bbd4-d3b9d3ef0660") : secret "plugin-serving-cert" not found
Jul 20 10:07:14.057700 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.057679    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.066571 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.066522    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pbw\" (UniqueName: \"kubernetes.io/projected/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-kube-api-access-97pbw\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.560585 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.560512    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.563069 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.563043    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4820f7c1-e06a-403e-bbd4-d3b9d3ef0660-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7fxgf\" (UID: \"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.836454 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.836372    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"
Jul 20 10:07:14.974527 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:14.974492    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf"]
Jul 20 10:07:14.977388 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:07:14.977360    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4820f7c1_e06a_403e_bbd4_d3b9d3ef0660.slice/crio-29c85ac45e71ebedc82752d7e887e0929ea26ff207cc0b313c97a8a471a7a78f WatchSource:0}: Error finding container 29c85ac45e71ebedc82752d7e887e0929ea26ff207cc0b313c97a8a471a7a78f: Status 404 returned error can't find the container with id 29c85ac45e71ebedc82752d7e887e0929ea26ff207cc0b313c97a8a471a7a78f
Jul 20 10:07:15.896967 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:15.896885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf" event={"ID":"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660","Type":"ContainerStarted","Data":"29c85ac45e71ebedc82752d7e887e0929ea26ff207cc0b313c97a8a471a7a78f"}
Jul 20 10:07:19.916383 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:19.916346    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf" event={"ID":"4820f7c1-e06a-403e-bbd4-d3b9d3ef0660","Type":"ContainerStarted","Data":"781b12e81dc209c66f90be066e2e4359a5e38ea49e678578fa371f415e3910f1"}
Jul 20 10:07:19.934255 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:19.934212    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7fxgf" podStartSLOduration=2.618244151 podStartE2EDuration="6.934198995s" totalImagesPullingTime="4.315954844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:19.933806415 +0000 UTC m=+619.946908312" watchObservedRunningTime="2026-07-20 10:07:19.934198995 +0000 UTC m=+619.947300891"
Jul 20 10:07:58.678573 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.678525    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-vv6k9"]
Jul 20 10:07:58.681653 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.681637    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-vv6k9"
Jul 20 10:07:58.684470 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.684454    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8cj5p\""
Jul 20 10:07:58.689470 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.689448    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-vv6k9"]
Jul 20 10:07:58.818666 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.818631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25qw2\" (UniqueName: \"kubernetes.io/projected/2a38647b-5e94-47fa-8dac-9f461a237e71-kube-api-access-25qw2\") pod \"authorino-b6c89bb85-vv6k9\" (UID: \"2a38647b-5e94-47fa-8dac-9f461a237e71\") " pod="kuadrant-system/authorino-b6c89bb85-vv6k9"
Jul 20 10:07:58.833235 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.833203    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-xnfr8"]
Jul 20 10:07:58.836199 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.836183    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-xnfr8"
Jul 20 10:07:58.844722 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.844698    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-xnfr8"]
Jul 20 10:07:58.919403 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.919375    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25qw2\" (UniqueName: \"kubernetes.io/projected/2a38647b-5e94-47fa-8dac-9f461a237e71-kube-api-access-25qw2\") pod \"authorino-b6c89bb85-vv6k9\" (UID: \"2a38647b-5e94-47fa-8dac-9f461a237e71\") " pod="kuadrant-system/authorino-b6c89bb85-vv6k9"
Jul 20 10:07:58.927898 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.927867    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25qw2\" (UniqueName: \"kubernetes.io/projected/2a38647b-5e94-47fa-8dac-9f461a237e71-kube-api-access-25qw2\") pod \"authorino-b6c89bb85-vv6k9\" (UID: \"2a38647b-5e94-47fa-8dac-9f461a237e71\") " pod="kuadrant-system/authorino-b6c89bb85-vv6k9"
Jul 20 10:07:58.990669 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:58.990644    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-vv6k9"
Jul 20 10:07:59.020474 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:59.020447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4hf\" (UniqueName: \"kubernetes.io/projected/e92bcee2-a1e3-45ef-a3df-1b2593c70d94-kube-api-access-pl4hf\") pod \"authorino-8df697c9f-xnfr8\" (UID: \"e92bcee2-a1e3-45ef-a3df-1b2593c70d94\") " pod="kuadrant-system/authorino-8df697c9f-xnfr8"
Jul 20 10:07:59.116436 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:59.116408    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-vv6k9"]
Jul 20 10:07:59.119149 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:07:59.119120    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a38647b_5e94_47fa_8dac_9f461a237e71.slice/crio-30772a7c510381b44e0fbad3b4a9a50daa012ba9daa729e7d03d81397edd4d80 WatchSource:0}: Error finding container 30772a7c510381b44e0fbad3b4a9a50daa012ba9daa729e7d03d81397edd4d80: Status 404 returned error can't find the container with id 30772a7c510381b44e0fbad3b4a9a50daa012ba9daa729e7d03d81397edd4d80
Jul 20 10:07:59.120876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:59.120847    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4hf\" (UniqueName: \"kubernetes.io/projected/e92bcee2-a1e3-45ef-a3df-1b2593c70d94-kube-api-access-pl4hf\") pod \"authorino-8df697c9f-xnfr8\" (UID: \"e92bcee2-a1e3-45ef-a3df-1b2593c70d94\") " pod="kuadrant-system/authorino-8df697c9f-xnfr8"
Jul 20 10:07:59.133059 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:59.133037    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4hf\" (UniqueName: \"kubernetes.io/projected/e92bcee2-a1e3-45ef-a3df-1b2593c70d94-kube-api-access-pl4hf\") pod \"authorino-8df697c9f-xnfr8\" (UID: \"e92bcee2-a1e3-45ef-a3df-1b2593c70d94\") " pod="kuadrant-system/authorino-8df697c9f-xnfr8"
Jul 20 10:07:59.144829 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:59.144806    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-xnfr8"
Jul 20 10:07:59.270321 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:07:59.270167    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-xnfr8"]
Jul 20 10:07:59.272744 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:07:59.272720    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode92bcee2_a1e3_45ef_a3df_1b2593c70d94.slice/crio-774a6dad0e342b2fbcb2f48235717407149c51e3d20bf1b1e03837e3a738a420 WatchSource:0}: Error finding container 774a6dad0e342b2fbcb2f48235717407149c51e3d20bf1b1e03837e3a738a420: Status 404 returned error can't find the container with id 774a6dad0e342b2fbcb2f48235717407149c51e3d20bf1b1e03837e3a738a420
Jul 20 10:08:00.071597 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:00.071528    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-xnfr8" event={"ID":"e92bcee2-a1e3-45ef-a3df-1b2593c70d94","Type":"ContainerStarted","Data":"774a6dad0e342b2fbcb2f48235717407149c51e3d20bf1b1e03837e3a738a420"}
Jul 20 10:08:00.073243 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:00.073213    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-vv6k9" event={"ID":"2a38647b-5e94-47fa-8dac-9f461a237e71","Type":"ContainerStarted","Data":"30772a7c510381b44e0fbad3b4a9a50daa012ba9daa729e7d03d81397edd4d80"}
Jul 20 10:08:02.083254 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:02.083216    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-xnfr8" event={"ID":"e92bcee2-a1e3-45ef-a3df-1b2593c70d94","Type":"ContainerStarted","Data":"396a1fa85e58eb80795fc55bdf3ad094db622029794ea637a736c32fedcca2d0"}
Jul 20 10:08:02.084497 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:02.084475    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-vv6k9" event={"ID":"2a38647b-5e94-47fa-8dac-9f461a237e71","Type":"ContainerStarted","Data":"8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521"}
Jul 20 10:08:02.106868 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:02.106820    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-xnfr8" podStartSLOduration=2.283435207 podStartE2EDuration="4.106806101s" totalImagesPullingTime="1.823370894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:02.105966438 +0000 UTC m=+662.119068335" watchObservedRunningTime="2026-07-20 10:08:02.106806101 +0000 UTC m=+662.119907996"
Jul 20 10:08:02.127919 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:02.127849    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-vv6k9" podStartSLOduration=2.1626480519999998 podStartE2EDuration="4.12782237s" totalImagesPullingTime="1.965174318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:02.126319287 +0000 UTC m=+662.139421184" watchObservedRunningTime="2026-07-20 10:08:02.12782237 +0000 UTC m=+662.140924280"
Jul 20 10:08:02.141788 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:02.141739    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-vv6k9"]
Jul 20 10:08:04.092119 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:04.092077    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-vv6k9" podUID="2a38647b-5e94-47fa-8dac-9f461a237e71" containerName="authorino" containerID="cri-o://8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521" gracePeriod=30
Jul 20 10:08:04.338544 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:04.338520    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-vv6k9"
Jul 20 10:08:04.357243 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:04.357173    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25qw2\" (UniqueName: \"kubernetes.io/projected/2a38647b-5e94-47fa-8dac-9f461a237e71-kube-api-access-25qw2\") pod \"2a38647b-5e94-47fa-8dac-9f461a237e71\" (UID: \"2a38647b-5e94-47fa-8dac-9f461a237e71\") "
Jul 20 10:08:04.359284 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:04.359256    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a38647b-5e94-47fa-8dac-9f461a237e71-kube-api-access-25qw2" (OuterVolumeSpecName: "kube-api-access-25qw2") pod "2a38647b-5e94-47fa-8dac-9f461a237e71" (UID: "2a38647b-5e94-47fa-8dac-9f461a237e71"). InnerVolumeSpecName "kube-api-access-25qw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:08:04.458305 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:04.458270    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25qw2\" (UniqueName: \"kubernetes.io/projected/2a38647b-5e94-47fa-8dac-9f461a237e71-kube-api-access-25qw2\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:08:05.096550 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.096515    2572 generic.go:356] "Generic (PLEG): container finished" podID="2a38647b-5e94-47fa-8dac-9f461a237e71" containerID="8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521" exitCode=0
Jul 20 10:08:05.096993 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.096586    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-vv6k9"
Jul 20 10:08:05.096993 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.096606    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-vv6k9" event={"ID":"2a38647b-5e94-47fa-8dac-9f461a237e71","Type":"ContainerDied","Data":"8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521"}
Jul 20 10:08:05.096993 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.096654    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-vv6k9" event={"ID":"2a38647b-5e94-47fa-8dac-9f461a237e71","Type":"ContainerDied","Data":"30772a7c510381b44e0fbad3b4a9a50daa012ba9daa729e7d03d81397edd4d80"}
Jul 20 10:08:05.096993 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.096677    2572 scope.go:117] "RemoveContainer" containerID="8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521"
Jul 20 10:08:05.104415 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.104400    2572 scope.go:117] "RemoveContainer" containerID="8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521"
Jul 20 10:08:05.104727 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:08:05.104705    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521\": container with ID starting with 8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521 not found: ID does not exist" containerID="8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521"
Jul 20 10:08:05.104816 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.104734    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521"} err="failed to get container status \"8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521\": rpc error: code = NotFound desc = could not find container \"8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521\": container with ID starting with 8ab26ea416dc8a66ec89696b1d976a6def3b857cd4e97f51973251b646c4e521 not found: ID does not exist"
Jul 20 10:08:05.117478 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.117458    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-vv6k9"]
Jul 20 10:08:05.123107 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:05.123089    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-vv6k9"]
Jul 20 10:08:06.519349 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:06.519320    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a38647b-5e94-47fa-8dac-9f461a237e71" path="/var/lib/kubelet/pods/2a38647b-5e94-47fa-8dac-9f461a237e71/volumes"
Jul 20 10:08:29.024200 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:29.024169    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-xnfr8"]
Jul 20 10:08:29.024633 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:29.024342    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-xnfr8" podUID="e92bcee2-a1e3-45ef-a3df-1b2593c70d94" containerName="authorino" containerID="cri-o://396a1fa85e58eb80795fc55bdf3ad094db622029794ea637a736c32fedcca2d0" gracePeriod=30
Jul 20 10:08:29.188399 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:29.188367    2572 generic.go:356] "Generic (PLEG): container finished" podID="e92bcee2-a1e3-45ef-a3df-1b2593c70d94" containerID="396a1fa85e58eb80795fc55bdf3ad094db622029794ea637a736c32fedcca2d0" exitCode=0
Jul 20 10:08:29.188533 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:29.188403    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-xnfr8" event={"ID":"e92bcee2-a1e3-45ef-a3df-1b2593c70d94","Type":"ContainerDied","Data":"396a1fa85e58eb80795fc55bdf3ad094db622029794ea637a736c32fedcca2d0"}
Jul 20 10:08:29.270730 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:29.270708    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-xnfr8"
Jul 20 10:08:29.340277 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:29.340194    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl4hf\" (UniqueName: \"kubernetes.io/projected/e92bcee2-a1e3-45ef-a3df-1b2593c70d94-kube-api-access-pl4hf\") pod \"e92bcee2-a1e3-45ef-a3df-1b2593c70d94\" (UID: \"e92bcee2-a1e3-45ef-a3df-1b2593c70d94\") "
Jul 20 10:08:29.342194 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:29.342156    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92bcee2-a1e3-45ef-a3df-1b2593c70d94-kube-api-access-pl4hf" (OuterVolumeSpecName: "kube-api-access-pl4hf") pod "e92bcee2-a1e3-45ef-a3df-1b2593c70d94" (UID: "e92bcee2-a1e3-45ef-a3df-1b2593c70d94"). InnerVolumeSpecName "kube-api-access-pl4hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:08:29.441189 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:29.441150    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl4hf\" (UniqueName: \"kubernetes.io/projected/e92bcee2-a1e3-45ef-a3df-1b2593c70d94-kube-api-access-pl4hf\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:08:30.193173 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:30.193139    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-xnfr8"
Jul 20 10:08:30.193173 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:30.193151    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-xnfr8" event={"ID":"e92bcee2-a1e3-45ef-a3df-1b2593c70d94","Type":"ContainerDied","Data":"774a6dad0e342b2fbcb2f48235717407149c51e3d20bf1b1e03837e3a738a420"}
Jul 20 10:08:30.193743 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:30.193201    2572 scope.go:117] "RemoveContainer" containerID="396a1fa85e58eb80795fc55bdf3ad094db622029794ea637a736c32fedcca2d0"
Jul 20 10:08:30.215702 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:30.215678    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-xnfr8"]
Jul 20 10:08:30.219129 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:30.219106    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-xnfr8"]
Jul 20 10:08:30.519970 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:30.519934    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92bcee2-a1e3-45ef-a3df-1b2593c70d94" path="/var/lib/kubelet/pods/e92bcee2-a1e3-45ef-a3df-1b2593c70d94/volumes"
Jul 20 10:08:32.294264 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.294228    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["observability/jaeger-6bbd7d4c9d-cgnvp"]
Jul 20 10:08:32.294685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.294523    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a38647b-5e94-47fa-8dac-9f461a237e71" containerName="authorino"
Jul 20 10:08:32.294685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.294535    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a38647b-5e94-47fa-8dac-9f461a237e71" containerName="authorino"
Jul 20 10:08:32.294685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.294548    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92bcee2-a1e3-45ef-a3df-1b2593c70d94" containerName="authorino"
Jul 20 10:08:32.294685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.294567    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92bcee2-a1e3-45ef-a3df-1b2593c70d94" containerName="authorino"
Jul 20 10:08:32.294685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.294622    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a38647b-5e94-47fa-8dac-9f461a237e71" containerName="authorino"
Jul 20 10:08:32.294685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.294630    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92bcee2-a1e3-45ef-a3df-1b2593c70d94" containerName="authorino"
Jul 20 10:08:32.298816 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.298799    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-cgnvp"
Jul 20 10:08:32.304188 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.304168    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"kube-root-ca.crt\""
Jul 20 10:08:32.304301 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.304235    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"observability\"/\"jaeger-dockercfg-rjhld\""
Jul 20 10:08:32.304938 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.304893    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"openshift-service-ca.crt\""
Jul 20 10:08:32.315910 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.315886    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-cgnvp"]
Jul 20 10:08:32.363842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.363809    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wm6j\" (UniqueName: \"kubernetes.io/projected/c83db330-c1c9-4d91-abcd-7ca2006295b3-kube-api-access-6wm6j\") pod \"jaeger-6bbd7d4c9d-cgnvp\" (UID: \"c83db330-c1c9-4d91-abcd-7ca2006295b3\") " pod="observability/jaeger-6bbd7d4c9d-cgnvp"
Jul 20 10:08:32.464863 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.464828    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wm6j\" (UniqueName: \"kubernetes.io/projected/c83db330-c1c9-4d91-abcd-7ca2006295b3-kube-api-access-6wm6j\") pod \"jaeger-6bbd7d4c9d-cgnvp\" (UID: \"c83db330-c1c9-4d91-abcd-7ca2006295b3\") " pod="observability/jaeger-6bbd7d4c9d-cgnvp"
Jul 20 10:08:32.474522 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.474494    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wm6j\" (UniqueName: \"kubernetes.io/projected/c83db330-c1c9-4d91-abcd-7ca2006295b3-kube-api-access-6wm6j\") pod \"jaeger-6bbd7d4c9d-cgnvp\" (UID: \"c83db330-c1c9-4d91-abcd-7ca2006295b3\") " pod="observability/jaeger-6bbd7d4c9d-cgnvp"
Jul 20 10:08:32.606975 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.606885    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-cgnvp"
Jul 20 10:08:32.737809 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.737789    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-cgnvp"]
Jul 20 10:08:32.740398 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:08:32.740363    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc83db330_c1c9_4d91_abcd_7ca2006295b3.slice/crio-23927c1beb6202a24e4d93c3776281032d371c355313dd9a9a7dc5ca54154863 WatchSource:0}: Error finding container 23927c1beb6202a24e4d93c3776281032d371c355313dd9a9a7dc5ca54154863: Status 404 returned error can't find the container with id 23927c1beb6202a24e4d93c3776281032d371c355313dd9a9a7dc5ca54154863
Jul 20 10:08:32.742244 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:32.742225    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:08:33.207307 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:33.207266    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-cgnvp" event={"ID":"c83db330-c1c9-4d91-abcd-7ca2006295b3","Type":"ContainerStarted","Data":"23927c1beb6202a24e4d93c3776281032d371c355313dd9a9a7dc5ca54154863"}
Jul 20 10:08:35.217255 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:35.217219    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-cgnvp" event={"ID":"c83db330-c1c9-4d91-abcd-7ca2006295b3","Type":"ContainerStarted","Data":"a4f8ebce40e2c82bb6779c62a74dc5eb84b7a37ec56bea38dc2b484f4ce424ad"}
Jul 20 10:08:35.217696 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:35.217424    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="observability/jaeger-6bbd7d4c9d-cgnvp"
Jul 20 10:08:35.219391 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:35.219372    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="observability/jaeger-6bbd7d4c9d-cgnvp"
Jul 20 10:08:35.239039 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:35.238999    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="observability/jaeger-6bbd7d4c9d-cgnvp" podStartSLOduration=1.338471892 podStartE2EDuration="3.238985698s" totalImagesPullingTime="1.900513806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:35.238496832 +0000 UTC m=+695.251598725" watchObservedRunningTime="2026-07-20 10:08:35.238985698 +0000 UTC m=+695.252087593"
Jul 20 10:08:37.878637 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:37.878597    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-l9pg7"]
Jul 20 10:08:37.879092 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:37.878787    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7" podUID="c83ecc8e-fa95-4b33-aa32-241c18b4e22e" containerName="manager" containerID="cri-o://07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d" gracePeriod=10
Jul 20 10:08:37.905429 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:37.905348    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-g46rm"]
Jul 20 10:08:37.908506 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:37.908485    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:37.915727 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:37.915706    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-g46rm"]
Jul 20 10:08:38.007281 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.007252    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4516489e-95a6-4df3-8a02-4f7deac66e56-cert\") pod \"kserve-controller-manager-645f4f9567-g46rm\" (UID: \"4516489e-95a6-4df3-8a02-4f7deac66e56\") " pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:38.007410 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.007295    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxz5l\" (UniqueName: \"kubernetes.io/projected/4516489e-95a6-4df3-8a02-4f7deac66e56-kube-api-access-gxz5l\") pod \"kserve-controller-manager-645f4f9567-g46rm\" (UID: \"4516489e-95a6-4df3-8a02-4f7deac66e56\") " pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:38.108035 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.107998    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxz5l\" (UniqueName: \"kubernetes.io/projected/4516489e-95a6-4df3-8a02-4f7deac66e56-kube-api-access-gxz5l\") pod \"kserve-controller-manager-645f4f9567-g46rm\" (UID: \"4516489e-95a6-4df3-8a02-4f7deac66e56\") " pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:38.108210 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.108123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4516489e-95a6-4df3-8a02-4f7deac66e56-cert\") pod \"kserve-controller-manager-645f4f9567-g46rm\" (UID: \"4516489e-95a6-4df3-8a02-4f7deac66e56\") " pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:38.110583 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.110532    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4516489e-95a6-4df3-8a02-4f7deac66e56-cert\") pod \"kserve-controller-manager-645f4f9567-g46rm\" (UID: \"4516489e-95a6-4df3-8a02-4f7deac66e56\") " pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:38.117011 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.116990    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxz5l\" (UniqueName: \"kubernetes.io/projected/4516489e-95a6-4df3-8a02-4f7deac66e56-kube-api-access-gxz5l\") pod \"kserve-controller-manager-645f4f9567-g46rm\" (UID: \"4516489e-95a6-4df3-8a02-4f7deac66e56\") " pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:38.129813 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.129766    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:08:38.209228 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.209197    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzlxv\" (UniqueName: \"kubernetes.io/projected/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-kube-api-access-bzlxv\") pod \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\" (UID: \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\") "
Jul 20 10:08:38.209384 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.209280    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-cert\") pod \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\" (UID: \"c83ecc8e-fa95-4b33-aa32-241c18b4e22e\") "
Jul 20 10:08:38.211266 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.211238    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-cert" (OuterVolumeSpecName: "cert") pod "c83ecc8e-fa95-4b33-aa32-241c18b4e22e" (UID: "c83ecc8e-fa95-4b33-aa32-241c18b4e22e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:08:38.211383 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.211287    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-kube-api-access-bzlxv" (OuterVolumeSpecName: "kube-api-access-bzlxv") pod "c83ecc8e-fa95-4b33-aa32-241c18b4e22e" (UID: "c83ecc8e-fa95-4b33-aa32-241c18b4e22e"). InnerVolumeSpecName "kube-api-access-bzlxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:08:38.229715 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.229691    2572 generic.go:356] "Generic (PLEG): container finished" podID="c83ecc8e-fa95-4b33-aa32-241c18b4e22e" containerID="07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d" exitCode=0
Jul 20 10:08:38.229827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.229730    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7"
Jul 20 10:08:38.229827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.229767    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7" event={"ID":"c83ecc8e-fa95-4b33-aa32-241c18b4e22e","Type":"ContainerDied","Data":"07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d"}
Jul 20 10:08:38.229827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.229822    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-l9pg7" event={"ID":"c83ecc8e-fa95-4b33-aa32-241c18b4e22e","Type":"ContainerDied","Data":"8f20a5c3fa0348e81985dfcecf37aeca52c613e386048832e15d3318b5651015"}
Jul 20 10:08:38.229927 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.229850    2572 scope.go:117] "RemoveContainer" containerID="07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d"
Jul 20 10:08:38.237764 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.237741    2572 scope.go:117] "RemoveContainer" containerID="07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d"
Jul 20 10:08:38.238054 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:08:38.238034    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d\": container with ID starting with 07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d not found: ID does not exist" containerID="07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d"
Jul 20 10:08:38.238131 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.238060    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d"} err="failed to get container status \"07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d\": rpc error: code = NotFound desc = could not find container \"07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d\": container with ID starting with 07d27dbe91cb12a22e4c4d567521e92f80d2879e31ff7a40ee2d226317ea416d not found: ID does not exist"
Jul 20 10:08:38.250862 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.250829    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:38.251271 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.251251    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-l9pg7"]
Jul 20 10:08:38.255328 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.255308    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-l9pg7"]
Jul 20 10:08:38.310803 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.310772    2572 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:08:38.310933 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.310806    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlxv\" (UniqueName: \"kubernetes.io/projected/c83ecc8e-fa95-4b33-aa32-241c18b4e22e-kube-api-access-bzlxv\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:08:38.381031 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.380956    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-g46rm"]
Jul 20 10:08:38.384051 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:08:38.384025    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4516489e_95a6_4df3_8a02_4f7deac66e56.slice/crio-6c8a27ac6be1c30c1fa9ab456a6b25b5d6aeeefdd0c5d2ce8ed6fb6995c325a7 WatchSource:0}: Error finding container 6c8a27ac6be1c30c1fa9ab456a6b25b5d6aeeefdd0c5d2ce8ed6fb6995c325a7: Status 404 returned error can't find the container with id 6c8a27ac6be1c30c1fa9ab456a6b25b5d6aeeefdd0c5d2ce8ed6fb6995c325a7
Jul 20 10:08:38.519604 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:38.519545    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83ecc8e-fa95-4b33-aa32-241c18b4e22e" path="/var/lib/kubelet/pods/c83ecc8e-fa95-4b33-aa32-241c18b4e22e/volumes"
Jul 20 10:08:39.236235 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:39.236200    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-g46rm" event={"ID":"4516489e-95a6-4df3-8a02-4f7deac66e56","Type":"ContainerStarted","Data":"89a559b5e3673dfd19c99440e70619c632ddd3dcd332d604baa4f5411a7f706f"}
Jul 20 10:08:39.236648 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:39.236262    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-g46rm" event={"ID":"4516489e-95a6-4df3-8a02-4f7deac66e56","Type":"ContainerStarted","Data":"6c8a27ac6be1c30c1fa9ab456a6b25b5d6aeeefdd0c5d2ce8ed6fb6995c325a7"}
Jul 20 10:08:39.236648 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:39.236312    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:08:39.258333 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:08:39.258286    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-645f4f9567-g46rm" podStartSLOduration=1.832239934 podStartE2EDuration="2.258271185s" totalImagesPullingTime="426.031251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:39.256109379 +0000 UTC m=+699.269211274" watchObservedRunningTime="2026-07-20 10:08:39.258271185 +0000 UTC m=+699.271373081"
Jul 20 10:09:10.244730 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:10.244689    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-645f4f9567-g46rm"
Jul 20 10:09:28.181830 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.181794    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2bmzn"]
Jul 20 10:09:28.182216 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.182094    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83ecc8e-fa95-4b33-aa32-241c18b4e22e" containerName="manager"
Jul 20 10:09:28.182216 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.182106    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83ecc8e-fa95-4b33-aa32-241c18b4e22e" containerName="manager"
Jul 20 10:09:28.182216 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.182155    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83ecc8e-fa95-4b33-aa32-241c18b4e22e" containerName="manager"
Jul 20 10:09:28.184839 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.184825    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2bmzn"
Jul 20 10:09:28.187958 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.187928    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 20 10:09:28.188082 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.188066    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7wwmc\""
Jul 20 10:09:28.193320 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.193300    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2bmzn"]
Jul 20 10:09:28.303299 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.303268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72cg6\" (UniqueName: \"kubernetes.io/projected/2dc82dae-2654-417e-b3a5-bcb4a46ff195-kube-api-access-72cg6\") pod \"s3-init-2bmzn\" (UID: \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\") " pod="kserve/s3-init-2bmzn"
Jul 20 10:09:28.303467 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.303306    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc82dae-2654-417e-b3a5-bcb4a46ff195-model-cache\") pod \"s3-init-2bmzn\" (UID: \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\") " pod="kserve/s3-init-2bmzn"
Jul 20 10:09:28.404689 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.404656    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72cg6\" (UniqueName: \"kubernetes.io/projected/2dc82dae-2654-417e-b3a5-bcb4a46ff195-kube-api-access-72cg6\") pod \"s3-init-2bmzn\" (UID: \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\") " pod="kserve/s3-init-2bmzn"
Jul 20 10:09:28.404689 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.404693    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc82dae-2654-417e-b3a5-bcb4a46ff195-model-cache\") pod \"s3-init-2bmzn\" (UID: \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\") " pod="kserve/s3-init-2bmzn"
Jul 20 10:09:28.405025 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.405011    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc82dae-2654-417e-b3a5-bcb4a46ff195-model-cache\") pod \"s3-init-2bmzn\" (UID: \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\") " pod="kserve/s3-init-2bmzn"
Jul 20 10:09:28.414367 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.414339    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72cg6\" (UniqueName: \"kubernetes.io/projected/2dc82dae-2654-417e-b3a5-bcb4a46ff195-kube-api-access-72cg6\") pod \"s3-init-2bmzn\" (UID: \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\") " pod="kserve/s3-init-2bmzn"
Jul 20 10:09:28.494099 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.494075    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2bmzn"
Jul 20 10:09:28.628065 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:28.628040    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2bmzn"]
Jul 20 10:09:28.630579 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:09:28.630540    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc82dae_2654_417e_b3a5_bcb4a46ff195.slice/crio-1657128fd32f2c9773cfe80de30a4076a9a91b3a839fb7900a89b2f78af9e33f WatchSource:0}: Error finding container 1657128fd32f2c9773cfe80de30a4076a9a91b3a839fb7900a89b2f78af9e33f: Status 404 returned error can't find the container with id 1657128fd32f2c9773cfe80de30a4076a9a91b3a839fb7900a89b2f78af9e33f
Jul 20 10:09:29.428998 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:29.428956    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2bmzn" event={"ID":"2dc82dae-2654-417e-b3a5-bcb4a46ff195","Type":"ContainerStarted","Data":"1657128fd32f2c9773cfe80de30a4076a9a91b3a839fb7900a89b2f78af9e33f"}
Jul 20 10:09:32.443820 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:32.443786    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2bmzn" event={"ID":"2dc82dae-2654-417e-b3a5-bcb4a46ff195","Type":"ContainerStarted","Data":"2c2cebdb1be4dec7c17162989aa1afadbb1c515c49178f7f190049624a04dc34"}
Jul 20 10:09:33.448855 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:33.448818    2572 generic.go:356] "Generic (PLEG): container finished" podID="2dc82dae-2654-417e-b3a5-bcb4a46ff195" containerID="2c2cebdb1be4dec7c17162989aa1afadbb1c515c49178f7f190049624a04dc34" exitCode=0
Jul 20 10:09:33.449301 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:33.448903    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2bmzn" event={"ID":"2dc82dae-2654-417e-b3a5-bcb4a46ff195","Type":"ContainerDied","Data":"2c2cebdb1be4dec7c17162989aa1afadbb1c515c49178f7f190049624a04dc34"}
Jul 20 10:09:38.471088 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:38.471050    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2bmzn" event={"ID":"2dc82dae-2654-417e-b3a5-bcb4a46ff195","Type":"ContainerStarted","Data":"d148a0fb232ddf5aebd0ee39fd756881dcc390356be26e1030da68578f156ca5"}
Jul 20 10:09:38.487996 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:09:38.487952    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2bmzn" podStartSLOduration=1.445947514 podStartE2EDuration="10.487937499s" totalImagesPullingTime="7.534271642s" totalInitContainerRuntime="1.507718343s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:38.487326537 +0000 UTC m=+758.500428430" watchObservedRunningTime="2026-07-20 10:09:38.487937499 +0000 UTC m=+758.501039393"
Jul 20 10:10:43.709165 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:43.709130    2572 generic.go:356] "Generic (PLEG): container finished" podID="2dc82dae-2654-417e-b3a5-bcb4a46ff195" containerID="d148a0fb232ddf5aebd0ee39fd756881dcc390356be26e1030da68578f156ca5" exitCode=0
Jul 20 10:10:43.709640 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:43.709214    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2bmzn" event={"ID":"2dc82dae-2654-417e-b3a5-bcb4a46ff195","Type":"ContainerDied","Data":"d148a0fb232ddf5aebd0ee39fd756881dcc390356be26e1030da68578f156ca5"}
Jul 20 10:10:44.842545 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:44.842520    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2bmzn"
Jul 20 10:10:44.920804 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:44.920767    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72cg6\" (UniqueName: \"kubernetes.io/projected/2dc82dae-2654-417e-b3a5-bcb4a46ff195-kube-api-access-72cg6\") pod \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\" (UID: \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\") "
Jul 20 10:10:44.920967 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:44.920833    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc82dae-2654-417e-b3a5-bcb4a46ff195-model-cache\") pod \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\" (UID: \"2dc82dae-2654-417e-b3a5-bcb4a46ff195\") "
Jul 20 10:10:44.922934 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:44.922907    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc82dae-2654-417e-b3a5-bcb4a46ff195-kube-api-access-72cg6" (OuterVolumeSpecName: "kube-api-access-72cg6") pod "2dc82dae-2654-417e-b3a5-bcb4a46ff195" (UID: "2dc82dae-2654-417e-b3a5-bcb4a46ff195"). InnerVolumeSpecName "kube-api-access-72cg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:10:44.943357 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:44.943323    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc82dae-2654-417e-b3a5-bcb4a46ff195-model-cache" (OuterVolumeSpecName: "model-cache") pod "2dc82dae-2654-417e-b3a5-bcb4a46ff195" (UID: "2dc82dae-2654-417e-b3a5-bcb4a46ff195"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:10:45.021543 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:45.021463    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc82dae-2654-417e-b3a5-bcb4a46ff195-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:10:45.021543 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:45.021494    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72cg6\" (UniqueName: \"kubernetes.io/projected/2dc82dae-2654-417e-b3a5-bcb4a46ff195-kube-api-access-72cg6\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:10:45.718253 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:45.718218    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2bmzn" event={"ID":"2dc82dae-2654-417e-b3a5-bcb4a46ff195","Type":"ContainerDied","Data":"1657128fd32f2c9773cfe80de30a4076a9a91b3a839fb7900a89b2f78af9e33f"}
Jul 20 10:10:45.718253 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:45.718239    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2bmzn"
Jul 20 10:10:45.718444 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:10:45.718262    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1657128fd32f2c9773cfe80de30a4076a9a91b3a839fb7900a89b2f78af9e33f"
Jul 20 10:11:11.211820 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.211783    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"]
Jul 20 10:11:11.212393 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.212202    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc82dae-2654-417e-b3a5-bcb4a46ff195" containerName="download-hf-model"
Jul 20 10:11:11.212393 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.212219    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc82dae-2654-417e-b3a5-bcb4a46ff195" containerName="download-hf-model"
Jul 20 10:11:11.212393 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.212239    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc82dae-2654-417e-b3a5-bcb4a46ff195" containerName="s3-init"
Jul 20 10:11:11.212393 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.212249    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc82dae-2654-417e-b3a5-bcb4a46ff195" containerName="s3-init"
Jul 20 10:11:11.212393 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.212331    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dc82dae-2654-417e-b3a5-bcb4a46ff195" containerName="s3-init"
Jul 20 10:11:11.215364 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.215343    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.219752 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.219730    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 20 10:11:11.221064 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.221047    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-8xm5l\""
Jul 20 10:11:11.221166 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.221050    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 20 10:11:11.221166 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.221051    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 20 10:11:11.226387 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.226364    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1faa8d-06be-4aa6-bad5-e838a4713400-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.226505 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.226408    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.226505 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.226462    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.226505 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.226486    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.226699 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.226521    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bv8\" (UniqueName: \"kubernetes.io/projected/be1faa8d-06be-4aa6-bad5-e838a4713400-kube-api-access-v9bv8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.226699 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.226599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.226699 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.226653    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"]
Jul 20 10:11:11.327772 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.327741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1faa8d-06be-4aa6-bad5-e838a4713400-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.327772 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.327776    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.328016 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.327795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.328016 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.327809    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.328016 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.327826    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bv8\" (UniqueName: \"kubernetes.io/projected/be1faa8d-06be-4aa6-bad5-e838a4713400-kube-api-access-v9bv8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.328016 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.327856    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.328220 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.328198    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.328274 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.328233    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.328312 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.328275    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.330338 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.330313    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.331122 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.331101    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1faa8d-06be-4aa6-bad5-e838a4713400-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.336725 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.336697    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bv8\" (UniqueName: \"kubernetes.io/projected/be1faa8d-06be-4aa6-bad5-e838a4713400-kube-api-access-v9bv8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z9vw2\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.526197 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.526090    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:11.660511 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.660486    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"]
Jul 20 10:11:11.662496 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:11:11.662465    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1faa8d_06be_4aa6_bad5_e838a4713400.slice/crio-9348db5a3ab8e5c31839dad5cf24e8106595add9a012f1cb65d7a758044e4660 WatchSource:0}: Error finding container 9348db5a3ab8e5c31839dad5cf24e8106595add9a012f1cb65d7a758044e4660: Status 404 returned error can't find the container with id 9348db5a3ab8e5c31839dad5cf24e8106595add9a012f1cb65d7a758044e4660
Jul 20 10:11:11.817867 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:11.817786    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2" event={"ID":"be1faa8d-06be-4aa6-bad5-e838a4713400","Type":"ContainerStarted","Data":"9348db5a3ab8e5c31839dad5cf24e8106595add9a012f1cb65d7a758044e4660"}
Jul 20 10:11:13.827743 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:13.827704    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2" event={"ID":"be1faa8d-06be-4aa6-bad5-e838a4713400","Type":"ContainerStarted","Data":"176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5"}
Jul 20 10:11:13.853373 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:13.853328    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2" podStartSLOduration=1.399455624 podStartE2EDuration="2.85331391s" totalImagesPullingTime="1.453858286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:11:13.851852488 +0000 UTC m=+853.864954383" watchObservedRunningTime="2026-07-20 10:11:13.85331391 +0000 UTC m=+853.866415811"
Jul 20 10:11:14.526256 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:14.526224    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:21.526501 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:21.526459    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:21.538791 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:21.538765    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:21.876542 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:21.876509    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:56.420743 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.420656    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"]
Jul 20 10:11:56.439190 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:11:56.439149    2572 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:11:56.439190 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:11:56.439190 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:11:56.439190 ip-10-0-138-196 kubenswrapper[2572]:  > containerID="176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5" cmd=["/bin/sleep","15"]
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.439210    2572 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2" message="" err=<
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]:  >
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:11:56.439223    2572 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2" podUID="be1faa8d-06be-4aa6-bad5-e838a4713400" containerName="main" containerID="cri-o://176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5"
Jul 20 10:11:56.439484 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.439280    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2" podUID="be1faa8d-06be-4aa6-bad5-e838a4713400" containerName="main" containerID="cri-o://176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5" gracePeriod=60
Jul 20 10:11:56.786051 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.786030    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:56.915716 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.915685    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-dshm\") pod \"be1faa8d-06be-4aa6-bad5-e838a4713400\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") "
Jul 20 10:11:56.915911 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.915750    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-model-cache\") pod \"be1faa8d-06be-4aa6-bad5-e838a4713400\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") "
Jul 20 10:11:56.915911 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.915817    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-home\") pod \"be1faa8d-06be-4aa6-bad5-e838a4713400\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") "
Jul 20 10:11:56.915911 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.915843    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9bv8\" (UniqueName: \"kubernetes.io/projected/be1faa8d-06be-4aa6-bad5-e838a4713400-kube-api-access-v9bv8\") pod \"be1faa8d-06be-4aa6-bad5-e838a4713400\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") "
Jul 20 10:11:56.915911 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.915878    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-tmp-dir\") pod \"be1faa8d-06be-4aa6-bad5-e838a4713400\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") "
Jul 20 10:11:56.915911 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.915907    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1faa8d-06be-4aa6-bad5-e838a4713400-tls-certs\") pod \"be1faa8d-06be-4aa6-bad5-e838a4713400\" (UID: \"be1faa8d-06be-4aa6-bad5-e838a4713400\") "
Jul 20 10:11:56.916143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.915974    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-model-cache" (OuterVolumeSpecName: "model-cache") pod "be1faa8d-06be-4aa6-bad5-e838a4713400" (UID: "be1faa8d-06be-4aa6-bad5-e838a4713400"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:11:56.916143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.916067    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-home" (OuterVolumeSpecName: "home") pod "be1faa8d-06be-4aa6-bad5-e838a4713400" (UID: "be1faa8d-06be-4aa6-bad5-e838a4713400"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:11:56.916143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.916136    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:11:56.916260 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.916136    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be1faa8d-06be-4aa6-bad5-e838a4713400" (UID: "be1faa8d-06be-4aa6-bad5-e838a4713400"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:11:56.916260 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.916149    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:11:56.917985 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.917960    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-dshm" (OuterVolumeSpecName: "dshm") pod "be1faa8d-06be-4aa6-bad5-e838a4713400" (UID: "be1faa8d-06be-4aa6-bad5-e838a4713400"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:11:56.918055 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.917982    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be1faa8d-06be-4aa6-bad5-e838a4713400-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be1faa8d-06be-4aa6-bad5-e838a4713400" (UID: "be1faa8d-06be-4aa6-bad5-e838a4713400"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:11:56.918055 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:56.918027    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1faa8d-06be-4aa6-bad5-e838a4713400-kube-api-access-v9bv8" (OuterVolumeSpecName: "kube-api-access-v9bv8") pod "be1faa8d-06be-4aa6-bad5-e838a4713400" (UID: "be1faa8d-06be-4aa6-bad5-e838a4713400"). InnerVolumeSpecName "kube-api-access-v9bv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:11:57.002532 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.002451    2572 generic.go:356] "Generic (PLEG): container finished" podID="be1faa8d-06be-4aa6-bad5-e838a4713400" containerID="176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5" exitCode=0
Jul 20 10:11:57.002532 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.002493    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"
Jul 20 10:11:57.002718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.002544    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2" event={"ID":"be1faa8d-06be-4aa6-bad5-e838a4713400","Type":"ContainerDied","Data":"176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5"}
Jul 20 10:11:57.002718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.002614    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2" event={"ID":"be1faa8d-06be-4aa6-bad5-e838a4713400","Type":"ContainerDied","Data":"9348db5a3ab8e5c31839dad5cf24e8106595add9a012f1cb65d7a758044e4660"}
Jul 20 10:11:57.002718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.002638    2572 scope.go:117] "RemoveContainer" containerID="176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5"
Jul 20 10:11:57.011156 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.011139    2572 scope.go:117] "RemoveContainer" containerID="176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5"
Jul 20 10:11:57.011460 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:11:57.011439    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5\": container with ID starting with 176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5 not found: ID does not exist" containerID="176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5"
Jul 20 10:11:57.011541 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.011465    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5"} err="failed to get container status \"176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5\": rpc error: code = NotFound desc = could not find container \"176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5\": container with ID starting with 176e21ac40e2a29262227566c92138ff421d80490dfa49a56775b0ddab3f53f5 not found: ID does not exist"
Jul 20 10:11:57.016842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.016824    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bv8\" (UniqueName: \"kubernetes.io/projected/be1faa8d-06be-4aa6-bad5-e838a4713400-kube-api-access-v9bv8\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:11:57.016842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.016842    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:11:57.016983 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.016852    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1faa8d-06be-4aa6-bad5-e838a4713400-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:11:57.016983 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.016860    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1faa8d-06be-4aa6-bad5-e838a4713400-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:11:57.035157 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.035132    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"]
Jul 20 10:11:57.043164 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:57.041201    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z9vw2"]
Jul 20 10:11:58.519887 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:11:58.519849    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be1faa8d-06be-4aa6-bad5-e838a4713400" path="/var/lib/kubelet/pods/be1faa8d-06be-4aa6-bad5-e838a4713400/volumes"
Jul 20 10:12:00.419496 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:00.419472    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:12:00.422213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:00.422193    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:12:23.227099 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.227063    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"]
Jul 20 10:12:23.227483 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.227340    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be1faa8d-06be-4aa6-bad5-e838a4713400" containerName="main"
Jul 20 10:12:23.227483 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.227351    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="be1faa8d-06be-4aa6-bad5-e838a4713400" containerName="main"
Jul 20 10:12:23.227483 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.227415    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="be1faa8d-06be-4aa6-bad5-e838a4713400" containerName="main"
Jul 20 10:12:23.231638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.231612    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.238861 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.238837    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-rbffz\""
Jul 20 10:12:23.238990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.238837    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 20 10:12:23.238990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.238842    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 20 10:12:23.238990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.238889    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 20 10:12:23.247037 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.247008    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"]
Jul 20 10:12:23.317438 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.317396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.317643 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.317444    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.317643 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.317468    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.317643 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.317584    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.317643 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.317619    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td7s5\" (UniqueName: \"kubernetes.io/projected/3b16fd28-f2de-4348-b6ec-15688fe38dc7-kube-api-access-td7s5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.317781 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.317652    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.418870 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.418820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.418870 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.418867    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td7s5\" (UniqueName: \"kubernetes.io/projected/3b16fd28-f2de-4348-b6ec-15688fe38dc7-kube-api-access-td7s5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.419114 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.418893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.419114 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.418923    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.419114 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.418949    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.419114 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.418968    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.419342 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.419318    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.419342 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.419333    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.419448 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.419370    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.421312 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.421292    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.421584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.421550    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.430142 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.430110    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td7s5\" (UniqueName: \"kubernetes.io/projected/3b16fd28-f2de-4348-b6ec-15688fe38dc7-kube-api-access-td7s5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.542197 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.542115    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:23.678593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.678548    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"]
Jul 20 10:12:23.680500 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:12:23.680475    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b16fd28_f2de_4348_b6ec_15688fe38dc7.slice/crio-d35383fcd85c8dc50e2114c578b85e6b309fe3b24fab4de02819ea6d129a139a WatchSource:0}: Error finding container d35383fcd85c8dc50e2114c578b85e6b309fe3b24fab4de02819ea6d129a139a: Status 404 returned error can't find the container with id d35383fcd85c8dc50e2114c578b85e6b309fe3b24fab4de02819ea6d129a139a
Jul 20 10:12:23.942989 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.942904    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"]
Jul 20 10:12:23.946131 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.946111    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:23.949191 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.949157    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-45kc6\""
Jul 20 10:12:23.958646 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:23.958620    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"]
Jul 20 10:12:24.024196 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.024156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/efa0190e-8078-4de4-aa28-90311b55cfb9-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.024391 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.024225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa0190e-8078-4de4-aa28-90311b55cfb9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.024391 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.024263    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td265\" (UniqueName: \"kubernetes.io/projected/efa0190e-8078-4de4-aa28-90311b55cfb9-kube-api-access-td265\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.111805 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.111771    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6" event={"ID":"3b16fd28-f2de-4348-b6ec-15688fe38dc7","Type":"ContainerStarted","Data":"b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6"}
Jul 20 10:12:24.112010 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.111828    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6" event={"ID":"3b16fd28-f2de-4348-b6ec-15688fe38dc7","Type":"ContainerStarted","Data":"d35383fcd85c8dc50e2114c578b85e6b309fe3b24fab4de02819ea6d129a139a"}
Jul 20 10:12:24.124724 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.124688    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td265\" (UniqueName: \"kubernetes.io/projected/efa0190e-8078-4de4-aa28-90311b55cfb9-kube-api-access-td265\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.124905 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.124776    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/efa0190e-8078-4de4-aa28-90311b55cfb9-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.124905 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.124825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa0190e-8078-4de4-aa28-90311b55cfb9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.125191 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.125154    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/efa0190e-8078-4de4-aa28-90311b55cfb9-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.127338 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.127315    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa0190e-8078-4de4-aa28-90311b55cfb9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.139726 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.139699    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td265\" (UniqueName: \"kubernetes.io/projected/efa0190e-8078-4de4-aa28-90311b55cfb9-kube-api-access-td265\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.142607 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.142546    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6" podStartSLOduration=1.142532368 podStartE2EDuration="1.142532368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:12:24.140363168 +0000 UTC m=+924.153465065" watchObservedRunningTime="2026-07-20 10:12:24.142532368 +0000 UTC m=+924.155634289"
Jul 20 10:12:24.255897 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.255859    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:24.391901 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.391878    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"]
Jul 20 10:12:24.394515 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:12:24.394490    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa0190e_8078_4de4_aa28_90311b55cfb9.slice/crio-d5260009385fc9b2f9b2d1a5ac05cff248c33901e06d1c4d0436c91e47144f8d WatchSource:0}: Error finding container d5260009385fc9b2f9b2d1a5ac05cff248c33901e06d1c4d0436c91e47144f8d: Status 404 returned error can't find the container with id d5260009385fc9b2f9b2d1a5ac05cff248c33901e06d1c4d0436c91e47144f8d
Jul 20 10:12:24.542729 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:24.542642    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:25.118026 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:25.117984    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh" event={"ID":"efa0190e-8078-4de4-aa28-90311b55cfb9","Type":"ContainerStarted","Data":"d5260009385fc9b2f9b2d1a5ac05cff248c33901e06d1c4d0436c91e47144f8d"}
Jul 20 10:12:27.129054 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:27.129016    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh" event={"ID":"efa0190e-8078-4de4-aa28-90311b55cfb9","Type":"ContainerStarted","Data":"12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575"}
Jul 20 10:12:27.129437 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:27.129092    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:12:27.146128 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:27.146084    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh" podStartSLOduration=1.876430652 podStartE2EDuration="4.146068007s" totalImagesPullingTime="2.269637355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:12:27.145251107 +0000 UTC m=+927.158353006" watchObservedRunningTime="2026-07-20 10:12:27.146068007 +0000 UTC m=+927.159169902"
Jul 20 10:12:33.542544 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:33.542508    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:33.554834 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:33.554809    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:34.177728 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:34.177698    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:12:58.140406 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:12:58.140377    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:14:03.206671 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.206630    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"]
Jul 20 10:14:03.210106 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.210083    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.215290 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.215267    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 20 10:14:03.215508 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.215483    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 20 10:14:03.215828 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.215803    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 20 10:14:03.215901 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.215803    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 20 10:14:03.215901 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.215853    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-wbrnh\""
Jul 20 10:14:03.223832 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.223809    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"]
Jul 20 10:14:03.308666 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.308633    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-dshm\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.308868 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.308681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tmp-dir\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.308868 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.308785    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-home\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.308868 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.308809    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tls-certs\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.308868 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.308832    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmjt\" (UniqueName: \"kubernetes.io/projected/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kube-api-access-hcmjt\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.309043 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.308932    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-model-cache\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.309043 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.308985    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kserve-provision-location\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410269 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410234    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-home\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410269 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410268    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tls-certs\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410523 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmjt\" (UniqueName: \"kubernetes.io/projected/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kube-api-access-hcmjt\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410523 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410310    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-model-cache\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410523 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kserve-provision-location\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410523 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-dshm\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410523 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410376    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tmp-dir\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410810    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-home\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410910 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tmp-dir\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410910 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410887    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kserve-provision-location\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.410996 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.410949    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-model-cache\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.412926 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.412900    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-dshm\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.413024 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.412941    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tls-certs\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.419341 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.419318    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmjt\" (UniqueName: \"kubernetes.io/projected/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kube-api-access-hcmjt\") pod \"auth-enabled-test-kserve-6c997987dd-r59r7\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.520465 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.520380    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:03.671052 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.671018    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"]
Jul 20 10:14:03.674344 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:14:03.674319    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e3988c_bb7d_4c82_bb5d_f8bb06dcd862.slice/crio-24ac124f1064b2f3e9063744cd047ca3b4bcc0770429eb4feaade4deb9776097 WatchSource:0}: Error finding container 24ac124f1064b2f3e9063744cd047ca3b4bcc0770429eb4feaade4deb9776097: Status 404 returned error can't find the container with id 24ac124f1064b2f3e9063744cd047ca3b4bcc0770429eb4feaade4deb9776097
Jul 20 10:14:03.678909 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.678887    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"]
Jul 20 10:14:03.682113 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.682095    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.684723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.684705    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-q7z5g\""
Jul 20 10:14:03.693723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.693701    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"]
Jul 20 10:14:03.814680 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.814583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.814680 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.814625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.814909 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.814687    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.814909 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.814737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjbz\" (UniqueName: \"kubernetes.io/projected/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kube-api-access-xdjbz\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.915718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.915675    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.915718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.915714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.915984 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.915747    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.915984 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.915795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjbz\" (UniqueName: \"kubernetes.io/projected/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kube-api-access-xdjbz\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.916109 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.916062    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.916162 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.916118    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.918103 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.918079    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.925584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.925545    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjbz\" (UniqueName: \"kubernetes.io/projected/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kube-api-access-xdjbz\") pod \"auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:03.994776 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:03.994732    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:04.139363 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:04.139329    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"]
Jul 20 10:14:04.144005 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:14:04.143977    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99ce9e9_0067_46b0_a74e_9b05501a3c02.slice/crio-81842c5dc29d5bceca62e33ca775ec21910792077d0458375b0f752a6c6d4348 WatchSource:0}: Error finding container 81842c5dc29d5bceca62e33ca775ec21910792077d0458375b0f752a6c6d4348: Status 404 returned error can't find the container with id 81842c5dc29d5bceca62e33ca775ec21910792077d0458375b0f752a6c6d4348
Jul 20 10:14:04.529691 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:04.529653    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc" event={"ID":"b99ce9e9-0067-46b0-a74e-9b05501a3c02","Type":"ContainerStarted","Data":"91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da"}
Jul 20 10:14:04.530148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:04.529706    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc" event={"ID":"b99ce9e9-0067-46b0-a74e-9b05501a3c02","Type":"ContainerStarted","Data":"81842c5dc29d5bceca62e33ca775ec21910792077d0458375b0f752a6c6d4348"}
Jul 20 10:14:04.531241 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:04.531212    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" event={"ID":"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862","Type":"ContainerStarted","Data":"407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf"}
Jul 20 10:14:04.531336 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:04.531261    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" event={"ID":"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862","Type":"ContainerStarted","Data":"24ac124f1064b2f3e9063744cd047ca3b4bcc0770429eb4feaade4deb9776097"}
Jul 20 10:14:05.537697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:05.537665    2572 generic.go:356] "Generic (PLEG): container finished" podID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" containerID="91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da" exitCode=0
Jul 20 10:14:05.538175 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:05.537759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc" event={"ID":"b99ce9e9-0067-46b0-a74e-9b05501a3c02","Type":"ContainerDied","Data":"91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da"}
Jul 20 10:14:06.544545 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:06.544497    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc" event={"ID":"b99ce9e9-0067-46b0-a74e-9b05501a3c02","Type":"ContainerStarted","Data":"64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789"}
Jul 20 10:14:06.544979 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:06.544696    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:06.585929 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:06.585870    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc" podStartSLOduration=2.888774415 podStartE2EDuration="3.585834408s" totalImagesPullingTime="0s" totalInitContainerRuntime="697.059993ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:14:06.583855734 +0000 UTC m=+1026.596957630" watchObservedRunningTime="2026-07-20 10:14:06.585834408 +0000 UTC m=+1026.598936308"
Jul 20 10:14:06.615300 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:06.615256    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"]
Jul 20 10:14:06.625458 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:14:06.625402    2572 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:14:06.625458 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:06.625458 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:06.625458 ip-10-0-138-196 kubenswrapper[2572]:  > containerID="b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6" cmd=["/bin/sleep","15"]
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:06.625500    2572 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6" message="" err=<
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]:  >
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:14:06.625516    2572 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6" podUID="3b16fd28-f2de-4348-b6ec-15688fe38dc7" containerName="main" containerID="cri-o://b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6"
Jul 20 10:14:06.625854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:06.625587    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6" podUID="3b16fd28-f2de-4348-b6ec-15688fe38dc7" containerName="main" containerID="cri-o://b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6" gracePeriod=60
Jul 20 10:14:06.641537 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:06.641498    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"]
Jul 20 10:14:06.999908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:06.999879    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:14:07.145179 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145087    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tls-certs\") pod \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") "
Jul 20 10:14:07.145179 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145124    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-model-cache\") pod \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") "
Jul 20 10:14:07.145402 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145201    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tmp-dir\") pod \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") "
Jul 20 10:14:07.145402 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145228    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-dshm\") pod \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") "
Jul 20 10:14:07.145402 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145269    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-home\") pod \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") "
Jul 20 10:14:07.145402 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145303    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td7s5\" (UniqueName: \"kubernetes.io/projected/3b16fd28-f2de-4348-b6ec-15688fe38dc7-kube-api-access-td7s5\") pod \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\" (UID: \"3b16fd28-f2de-4348-b6ec-15688fe38dc7\") "
Jul 20 10:14:07.145623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145433    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b16fd28-f2de-4348-b6ec-15688fe38dc7" (UID: "3b16fd28-f2de-4348-b6ec-15688fe38dc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:07.145623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145448    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b16fd28-f2de-4348-b6ec-15688fe38dc7" (UID: "3b16fd28-f2de-4348-b6ec-15688fe38dc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:07.145623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145525    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-home" (OuterVolumeSpecName: "home") pod "3b16fd28-f2de-4348-b6ec-15688fe38dc7" (UID: "3b16fd28-f2de-4348-b6ec-15688fe38dc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:07.145623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145604    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.145623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.145624    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.147482 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.147460    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b16fd28-f2de-4348-b6ec-15688fe38dc7-kube-api-access-td7s5" (OuterVolumeSpecName: "kube-api-access-td7s5") pod "3b16fd28-f2de-4348-b6ec-15688fe38dc7" (UID: "3b16fd28-f2de-4348-b6ec-15688fe38dc7"). InnerVolumeSpecName "kube-api-access-td7s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:14:07.147661 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.147645    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b16fd28-f2de-4348-b6ec-15688fe38dc7" (UID: "3b16fd28-f2de-4348-b6ec-15688fe38dc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:14:07.147723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.147680    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-dshm" (OuterVolumeSpecName: "dshm") pod "3b16fd28-f2de-4348-b6ec-15688fe38dc7" (UID: "3b16fd28-f2de-4348-b6ec-15688fe38dc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:07.246460 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.246427    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b16fd28-f2de-4348-b6ec-15688fe38dc7-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.246460 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.246455    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.246460 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.246464    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b16fd28-f2de-4348-b6ec-15688fe38dc7-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.246694 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.246473    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td7s5\" (UniqueName: \"kubernetes.io/projected/3b16fd28-f2de-4348-b6ec-15688fe38dc7-kube-api-access-td7s5\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.550820 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.550788    2572 generic.go:356] "Generic (PLEG): container finished" podID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerID="407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf" exitCode=0
Jul 20 10:14:07.551248 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.550861    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" event={"ID":"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862","Type":"ContainerDied","Data":"407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf"}
Jul 20 10:14:07.552040 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.552020    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:14:07.552314 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.552291    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b16fd28-f2de-4348-b6ec-15688fe38dc7" containerID="b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6" exitCode=0
Jul 20 10:14:07.552409 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.552368    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6" event={"ID":"3b16fd28-f2de-4348-b6ec-15688fe38dc7","Type":"ContainerDied","Data":"b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6"}
Jul 20 10:14:07.552477 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.552424    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6" event={"ID":"3b16fd28-f2de-4348-b6ec-15688fe38dc7","Type":"ContainerDied","Data":"d35383fcd85c8dc50e2114c578b85e6b309fe3b24fab4de02819ea6d129a139a"}
Jul 20 10:14:07.552477 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.552373    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"
Jul 20 10:14:07.552477 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.552457    2572 scope.go:117] "RemoveContainer" containerID="b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6"
Jul 20 10:14:07.561609 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.561592    2572 scope.go:117] "RemoveContainer" containerID="b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6"
Jul 20 10:14:07.561854 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:14:07.561835    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6\": container with ID starting with b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6 not found: ID does not exist" containerID="b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6"
Jul 20 10:14:07.561956 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.561865    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6"} err="failed to get container status \"b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6\": rpc error: code = NotFound desc = could not find container \"b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6\": container with ID starting with b8bf16669bce037557d4a7050aa08e1926185e18eefa9ffbd4a29266b30f94e6 not found: ID does not exist"
Jul 20 10:14:07.613053 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.613019    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"]
Jul 20 10:14:07.626403 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:07.626381    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-q6xn6"]
Jul 20 10:14:08.141624 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:08.141329    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh" podUID="efa0190e-8078-4de4-aa28-90311b55cfb9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:14:08.523470 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:08.523038    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b16fd28-f2de-4348-b6ec-15688fe38dc7" path="/var/lib/kubelet/pods/3b16fd28-f2de-4348-b6ec-15688fe38dc7/volumes"
Jul 20 10:14:18.140214 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:18.140170    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh" podUID="efa0190e-8078-4de4-aa28-90311b55cfb9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:14:21.663525 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:21.663489    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh" podUID="efa0190e-8078-4de4-aa28-90311b55cfb9" containerName="main" containerID="cri-o://12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575" gracePeriod=45
Jul 20 10:14:21.974180 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:21.974151    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:14:22.089990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.089960    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td265\" (UniqueName: \"kubernetes.io/projected/efa0190e-8078-4de4-aa28-90311b55cfb9-kube-api-access-td265\") pod \"efa0190e-8078-4de4-aa28-90311b55cfb9\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") "
Jul 20 10:14:22.090181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.090022    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/efa0190e-8078-4de4-aa28-90311b55cfb9-tokenizer-uds\") pod \"efa0190e-8078-4de4-aa28-90311b55cfb9\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") "
Jul 20 10:14:22.090253 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.090200    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa0190e-8078-4de4-aa28-90311b55cfb9-tls-certs\") pod \"efa0190e-8078-4de4-aa28-90311b55cfb9\" (UID: \"efa0190e-8078-4de4-aa28-90311b55cfb9\") "
Jul 20 10:14:22.090309 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.090283    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa0190e-8078-4de4-aa28-90311b55cfb9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "efa0190e-8078-4de4-aa28-90311b55cfb9" (UID: "efa0190e-8078-4de4-aa28-90311b55cfb9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:22.090469 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.090447    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/efa0190e-8078-4de4-aa28-90311b55cfb9-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:22.092086 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.092060    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa0190e-8078-4de4-aa28-90311b55cfb9-kube-api-access-td265" (OuterVolumeSpecName: "kube-api-access-td265") pod "efa0190e-8078-4de4-aa28-90311b55cfb9" (UID: "efa0190e-8078-4de4-aa28-90311b55cfb9"). InnerVolumeSpecName "kube-api-access-td265". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:14:22.092176 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.092155    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa0190e-8078-4de4-aa28-90311b55cfb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efa0190e-8078-4de4-aa28-90311b55cfb9" (UID: "efa0190e-8078-4de4-aa28-90311b55cfb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:14:22.191508 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.191426    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa0190e-8078-4de4-aa28-90311b55cfb9-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:22.191508 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.191458    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td265\" (UniqueName: \"kubernetes.io/projected/efa0190e-8078-4de4-aa28-90311b55cfb9-kube-api-access-td265\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:14:22.641907 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.641870    2572 generic.go:356] "Generic (PLEG): container finished" podID="efa0190e-8078-4de4-aa28-90311b55cfb9" containerID="12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575" exitCode=0
Jul 20 10:14:22.642091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.641924    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"
Jul 20 10:14:22.642091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.641945    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh" event={"ID":"efa0190e-8078-4de4-aa28-90311b55cfb9","Type":"ContainerDied","Data":"12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575"}
Jul 20 10:14:22.642091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.641997    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh" event={"ID":"efa0190e-8078-4de4-aa28-90311b55cfb9","Type":"ContainerDied","Data":"d5260009385fc9b2f9b2d1a5ac05cff248c33901e06d1c4d0436c91e47144f8d"}
Jul 20 10:14:22.642091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.642029    2572 scope.go:117] "RemoveContainer" containerID="12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575"
Jul 20 10:14:22.651418 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.651393    2572 scope.go:117] "RemoveContainer" containerID="12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575"
Jul 20 10:14:22.651823 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:14:22.651800    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575\": container with ID starting with 12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575 not found: ID does not exist" containerID="12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575"
Jul 20 10:14:22.651921 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.651828    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575"} err="failed to get container status \"12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575\": rpc error: code = NotFound desc = could not find container \"12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575\": container with ID starting with 12c02a42ec870d5a1ec8cbfad155b2fc3970f46c1035559866248bc3fd74e575 not found: ID does not exist"
Jul 20 10:14:22.662321 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.662296    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"]
Jul 20 10:14:22.666292 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:22.666274    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-557wmlh"]
Jul 20 10:14:24.519085 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:24.519053    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efa0190e-8078-4de4-aa28-90311b55cfb9" path="/var/lib/kubelet/pods/efa0190e-8078-4de4-aa28-90311b55cfb9/volumes"
Jul 20 10:14:30.895766 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.895733    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"]
Jul 20 10:14:30.896162 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.896076    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa0190e-8078-4de4-aa28-90311b55cfb9" containerName="main"
Jul 20 10:14:30.896162 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.896093    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa0190e-8078-4de4-aa28-90311b55cfb9" containerName="main"
Jul 20 10:14:30.896162 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.896113    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b16fd28-f2de-4348-b6ec-15688fe38dc7" containerName="main"
Jul 20 10:14:30.896162 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.896122    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b16fd28-f2de-4348-b6ec-15688fe38dc7" containerName="main"
Jul 20 10:14:30.896310 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.896203    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b16fd28-f2de-4348-b6ec-15688fe38dc7" containerName="main"
Jul 20 10:14:30.896310 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.896213    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa0190e-8078-4de4-aa28-90311b55cfb9" containerName="main"
Jul 20 10:14:30.936403 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.936357    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"]
Jul 20 10:14:30.936607 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.936511    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:30.941222 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.941194    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 20 10:14:30.941363 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.941263    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 20 10:14:30.941363 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.941317    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-fxpnp\""
Jul 20 10:14:30.941363 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.941261    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-gs95k\""
Jul 20 10:14:30.941605 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.941586    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 20 10:14:30.966164 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.966141    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:30.966263 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.966179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:30.966312 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:30.966268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5t9x\" (UniqueName: \"kubernetes.io/projected/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-kube-api-access-h5t9x\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:31.067360 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:31.067325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5t9x\" (UniqueName: \"kubernetes.io/projected/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-kube-api-access-h5t9x\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:31.067537 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:31.067378    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:31.067537 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:31.067408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:31.067864 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:31.067843    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:31.070264 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:31.070236    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:31.077771 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:31.077727    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5t9x\" (UniqueName: \"kubernetes.io/projected/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-kube-api-access-h5t9x\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:31.247888 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:31.247853    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:37.555005 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:37.554969    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:14:38.343495 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:38.343464    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"]
Jul 20 10:14:38.344853 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:14:38.344826    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa97ff2b_e388_4bfe_8222_6f62b8c7e123.slice/crio-52c32c475332521b5a1e68abfcc6f88582c15075e147a9a8ed46dc5c2817a248 WatchSource:0}: Error finding container 52c32c475332521b5a1e68abfcc6f88582c15075e147a9a8ed46dc5c2817a248: Status 404 returned error can't find the container with id 52c32c475332521b5a1e68abfcc6f88582c15075e147a9a8ed46dc5c2817a248
Jul 20 10:14:38.722109 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:38.722073    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn" event={"ID":"fa97ff2b-e388-4bfe-8222-6f62b8c7e123","Type":"ContainerStarted","Data":"0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181"}
Jul 20 10:14:38.722464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:38.722153    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn" event={"ID":"fa97ff2b-e388-4bfe-8222-6f62b8c7e123","Type":"ContainerStarted","Data":"52c32c475332521b5a1e68abfcc6f88582c15075e147a9a8ed46dc5c2817a248"}
Jul 20 10:14:38.722464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:38.722273    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:14:38.761345 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:38.761293    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn" podStartSLOduration=8.761276431 podStartE2EDuration="8.761276431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:14:38.757173793 +0000 UTC m=+1058.770275689" watchObservedRunningTime="2026-07-20 10:14:38.761276431 +0000 UTC m=+1058.774378327"
Jul 20 10:14:39.728196 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:39.728163    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" event={"ID":"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862","Type":"ContainerStarted","Data":"3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090"}
Jul 20 10:14:39.757873 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:39.757815    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podStartSLOduration=2.217426946 podStartE2EDuration="36.75779652s" totalImagesPullingTime="31.079663088s" totalInitContainerRuntime="3.460706486s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:14:39.755217291 +0000 UTC m=+1059.768319200" watchObservedRunningTime="2026-07-20 10:14:39.75779652 +0000 UTC m=+1059.770898417"
Jul 20 10:14:40.520922 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:40.520887    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:43.521216 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:43.521178    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:14:43.522537 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:43.522508    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:14:53.521352 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:14:53.521262    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:15:03.521094 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:03.521048    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:15:09.730527 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:09.730496    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:15:11.412441 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:11.412406    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"]
Jul 20 10:15:13.520820 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:13.520774    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:15:19.730166 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:19.730125    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn" podUID="fa97ff2b-e388-4bfe-8222-6f62b8c7e123" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:15:23.521111 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:23.521069    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:15:26.433164 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.433092    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn" podUID="fa97ff2b-e388-4bfe-8222-6f62b8c7e123" containerName="main" containerID="cri-o://0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181" gracePeriod=45
Jul 20 10:15:26.700464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.700439    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:15:26.738486 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.738448    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tokenizer-uds\") pod \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") "
Jul 20 10:15:26.738680 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.738539    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5t9x\" (UniqueName: \"kubernetes.io/projected/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-kube-api-access-h5t9x\") pod \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") "
Jul 20 10:15:26.738680 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.738585    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tls-certs\") pod \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\" (UID: \"fa97ff2b-e388-4bfe-8222-6f62b8c7e123\") "
Jul 20 10:15:26.738804 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.738777    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "fa97ff2b-e388-4bfe-8222-6f62b8c7e123" (UID: "fa97ff2b-e388-4bfe-8222-6f62b8c7e123"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:26.740876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.740848    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa97ff2b-e388-4bfe-8222-6f62b8c7e123" (UID: "fa97ff2b-e388-4bfe-8222-6f62b8c7e123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:15:26.740876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.740846    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-kube-api-access-h5t9x" (OuterVolumeSpecName: "kube-api-access-h5t9x") pod "fa97ff2b-e388-4bfe-8222-6f62b8c7e123" (UID: "fa97ff2b-e388-4bfe-8222-6f62b8c7e123"). InnerVolumeSpecName "kube-api-access-h5t9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:15:26.839397 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.839361    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5t9x\" (UniqueName: \"kubernetes.io/projected/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-kube-api-access-h5t9x\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:15:26.839397 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.839391    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:15:26.839397 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.839401    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa97ff2b-e388-4bfe-8222-6f62b8c7e123-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:15:26.947656 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.947537    2572 generic.go:356] "Generic (PLEG): container finished" podID="fa97ff2b-e388-4bfe-8222-6f62b8c7e123" containerID="0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181" exitCode=0
Jul 20 10:15:26.947656 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.947594    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"
Jul 20 10:15:26.947656 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.947623    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn" event={"ID":"fa97ff2b-e388-4bfe-8222-6f62b8c7e123","Type":"ContainerDied","Data":"0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181"}
Jul 20 10:15:26.947924 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.947672    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn" event={"ID":"fa97ff2b-e388-4bfe-8222-6f62b8c7e123","Type":"ContainerDied","Data":"52c32c475332521b5a1e68abfcc6f88582c15075e147a9a8ed46dc5c2817a248"}
Jul 20 10:15:26.947924 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.947696    2572 scope.go:117] "RemoveContainer" containerID="0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181"
Jul 20 10:15:26.962409 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.962385    2572 scope.go:117] "RemoveContainer" containerID="0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181"
Jul 20 10:15:26.962764 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:15:26.962741    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181\": container with ID starting with 0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181 not found: ID does not exist" containerID="0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181"
Jul 20 10:15:26.962855 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.962776    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181"} err="failed to get container status \"0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181\": rpc error: code = NotFound desc = could not find container \"0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181\": container with ID starting with 0d56ec3c793584dc4c3f3b0abc6e2adc11d449c302c4a5a282c9133e27fcc181 not found: ID does not exist"
Jul 20 10:15:26.989267 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.989229    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"]
Jul 20 10:15:26.993279 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:26.993251    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched4j6hn"]
Jul 20 10:15:28.519904 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:28.519870    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa97ff2b-e388-4bfe-8222-6f62b8c7e123" path="/var/lib/kubelet/pods/fa97ff2b-e388-4bfe-8222-6f62b8c7e123/volumes"
Jul 20 10:15:33.521211 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:33.521178    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:15:43.521750 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:43.521700    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:15:53.521631 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:15:53.521583    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:16:03.521713 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:16:03.521665    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:16:13.521697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:16:13.521657    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:16:23.521528 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:16:23.521426    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:16:33.531471 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:16:33.531439    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:16:33.539924 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:16:33.539898    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:17:00.448551 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:00.448516    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:17:00.455374 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:00.455353    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:17:41.765845 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.765813    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"]
Jul 20 10:17:41.766235 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.766125    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa97ff2b-e388-4bfe-8222-6f62b8c7e123" containerName="main"
Jul 20 10:17:41.766235 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.766135    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa97ff2b-e388-4bfe-8222-6f62b8c7e123" containerName="main"
Jul 20 10:17:41.766235 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.766185    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa97ff2b-e388-4bfe-8222-6f62b8c7e123" containerName="main"
Jul 20 10:17:41.768590 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.768549    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.772786 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.772754    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 20 10:17:41.772786 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.772768    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 20 10:17:41.772786 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.772759    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-m5lgw\""
Jul 20 10:17:41.773041 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.772795    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-bnw6v\""
Jul 20 10:17:41.773041 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.772754    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 20 10:17:41.778511 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.778491    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"]
Jul 20 10:17:41.869492 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.869459    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm2gf\" (UniqueName: \"kubernetes.io/projected/a615cc8e-35a0-4220-91c2-f25f710b24fa-kube-api-access-xm2gf\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.869683 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.869508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a615cc8e-35a0-4220-91c2-f25f710b24fa-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.869683 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.869543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a615cc8e-35a0-4220-91c2-f25f710b24fa-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.970893 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.970854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a615cc8e-35a0-4220-91c2-f25f710b24fa-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.971079 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.971025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm2gf\" (UniqueName: \"kubernetes.io/projected/a615cc8e-35a0-4220-91c2-f25f710b24fa-kube-api-access-xm2gf\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.971146 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.971108    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a615cc8e-35a0-4220-91c2-f25f710b24fa-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.971347 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.971298    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a615cc8e-35a0-4220-91c2-f25f710b24fa-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.973495 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.973468    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a615cc8e-35a0-4220-91c2-f25f710b24fa-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:41.980862 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:41.980842    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm2gf\" (UniqueName: \"kubernetes.io/projected/a615cc8e-35a0-4220-91c2-f25f710b24fa-kube-api-access-xm2gf\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:42.079055 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:42.078965    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:42.223665 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:42.223635    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"]
Jul 20 10:17:42.224306 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:17:42.224282    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda615cc8e_35a0_4220_91c2_f25f710b24fa.slice/crio-26ff3ecc352bba9e3f05d24379f6795cb7064484f57d2ed74757d2aa4c507950 WatchSource:0}: Error finding container 26ff3ecc352bba9e3f05d24379f6795cb7064484f57d2ed74757d2aa4c507950: Status 404 returned error can't find the container with id 26ff3ecc352bba9e3f05d24379f6795cb7064484f57d2ed74757d2aa4c507950
Jul 20 10:17:42.519710 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:42.519676    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:17:42.519880 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:42.519732    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz" event={"ID":"a615cc8e-35a0-4220-91c2-f25f710b24fa","Type":"ContainerStarted","Data":"aedadc97257fc6422c11ebb02d06f5c9cd1963d927d7e8c3b4dbf157cc385b93"}
Jul 20 10:17:42.519880 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:42.519760    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz" event={"ID":"a615cc8e-35a0-4220-91c2-f25f710b24fa","Type":"ContainerStarted","Data":"26ff3ecc352bba9e3f05d24379f6795cb7064484f57d2ed74757d2aa4c507950"}
Jul 20 10:17:42.544357 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:17:42.544313    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz" podStartSLOduration=1.5442974619999998 podStartE2EDuration="1.544297462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:17:42.542137153 +0000 UTC m=+1242.555239051" watchObservedRunningTime="2026-07-20 10:17:42.544297462 +0000 UTC m=+1242.557399359"
Jul 20 10:18:13.525273 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:13.525242    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:18:15.465793 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:15.465760    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"]
Jul 20 10:18:23.525602 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:23.525547    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz" podUID="a615cc8e-35a0-4220-91c2-f25f710b24fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:18:30.485810 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.485768    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz" podUID="a615cc8e-35a0-4220-91c2-f25f710b24fa" containerName="main" containerID="cri-o://aedadc97257fc6422c11ebb02d06f5c9cd1963d927d7e8c3b4dbf157cc385b93" gracePeriod=45
Jul 20 10:18:30.721966 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.721934    2572 generic.go:356] "Generic (PLEG): container finished" podID="a615cc8e-35a0-4220-91c2-f25f710b24fa" containerID="aedadc97257fc6422c11ebb02d06f5c9cd1963d927d7e8c3b4dbf157cc385b93" exitCode=0
Jul 20 10:18:30.722105 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.721991    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz" event={"ID":"a615cc8e-35a0-4220-91c2-f25f710b24fa","Type":"ContainerDied","Data":"aedadc97257fc6422c11ebb02d06f5c9cd1963d927d7e8c3b4dbf157cc385b93"}
Jul 20 10:18:30.722105 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.722097    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz" event={"ID":"a615cc8e-35a0-4220-91c2-f25f710b24fa","Type":"ContainerDied","Data":"26ff3ecc352bba9e3f05d24379f6795cb7064484f57d2ed74757d2aa4c507950"}
Jul 20 10:18:30.722208 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.722125    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26ff3ecc352bba9e3f05d24379f6795cb7064484f57d2ed74757d2aa4c507950"
Jul 20 10:18:30.737911 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.737852    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:18:30.795061 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.795034    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm2gf\" (UniqueName: \"kubernetes.io/projected/a615cc8e-35a0-4220-91c2-f25f710b24fa-kube-api-access-xm2gf\") pod \"a615cc8e-35a0-4220-91c2-f25f710b24fa\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") "
Jul 20 10:18:30.795241 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.795108    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a615cc8e-35a0-4220-91c2-f25f710b24fa-tls-certs\") pod \"a615cc8e-35a0-4220-91c2-f25f710b24fa\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") "
Jul 20 10:18:30.795241 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.795137    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a615cc8e-35a0-4220-91c2-f25f710b24fa-tokenizer-uds\") pod \"a615cc8e-35a0-4220-91c2-f25f710b24fa\" (UID: \"a615cc8e-35a0-4220-91c2-f25f710b24fa\") "
Jul 20 10:18:30.795461 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.795440    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a615cc8e-35a0-4220-91c2-f25f710b24fa-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a615cc8e-35a0-4220-91c2-f25f710b24fa" (UID: "a615cc8e-35a0-4220-91c2-f25f710b24fa"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:30.797422 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.797383    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a615cc8e-35a0-4220-91c2-f25f710b24fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a615cc8e-35a0-4220-91c2-f25f710b24fa" (UID: "a615cc8e-35a0-4220-91c2-f25f710b24fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:18:30.797528 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.797443    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a615cc8e-35a0-4220-91c2-f25f710b24fa-kube-api-access-xm2gf" (OuterVolumeSpecName: "kube-api-access-xm2gf") pod "a615cc8e-35a0-4220-91c2-f25f710b24fa" (UID: "a615cc8e-35a0-4220-91c2-f25f710b24fa"). InnerVolumeSpecName "kube-api-access-xm2gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:18:30.895676 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.895640    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm2gf\" (UniqueName: \"kubernetes.io/projected/a615cc8e-35a0-4220-91c2-f25f710b24fa-kube-api-access-xm2gf\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:30.895676 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.895668    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a615cc8e-35a0-4220-91c2-f25f710b24fa-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:30.895676 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:30.895681    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a615cc8e-35a0-4220-91c2-f25f710b24fa-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:31.726755 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:31.726725    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"
Jul 20 10:18:31.747180 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:31.747156    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"]
Jul 20 10:18:31.750437 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:31.750415    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schegk4cz"]
Jul 20 10:18:32.519950 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:32.519915    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a615cc8e-35a0-4220-91c2-f25f710b24fa" path="/var/lib/kubelet/pods/a615cc8e-35a0-4220-91c2-f25f710b24fa/volumes"
Jul 20 10:18:36.301519 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:36.301490    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-auth-enabled-requires-token-d4ed2c08_auth-enabled-test-kserve-6c997987dd-r59r7_55e3988c-bb7d-4c82-bb5d-f8bb06dcd862/storage-initializer/0.log"
Jul 20 10:18:36.311611 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:36.311589    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-auth-enabled-requires-token-d4ed2c08_auth-enabled-test-kserve-6c997987dd-r59r7_55e3988c-bb7d-4c82-bb5d-f8bb06dcd862/main/0.log"
Jul 20 10:18:36.319440 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:36.319420    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-auth-enabled-requires-token-d4ed2c08_auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc_b99ce9e9-0067-46b0-a74e-9b05501a3c02/storage-initializer/0.log"
Jul 20 10:18:36.328684 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:36.328658    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-auth-enabled-requires-token-d4ed2c08_auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc_b99ce9e9-0067-46b0-a74e-9b05501a3c02/main/0.log"
Jul 20 10:18:39.285938 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.285898    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"]
Jul 20 10:18:39.286392 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.286326    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a615cc8e-35a0-4220-91c2-f25f710b24fa" containerName="main"
Jul 20 10:18:39.286392 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.286344    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a615cc8e-35a0-4220-91c2-f25f710b24fa" containerName="main"
Jul 20 10:18:39.286503 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.286444    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a615cc8e-35a0-4220-91c2-f25f710b24fa" containerName="main"
Jul 20 10:18:39.289785 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.289760    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.297353 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.297332    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 20 10:18:39.297466 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.297337    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 20 10:18:39.297466 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.297442    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-mx7hf\""
Jul 20 10:18:39.297466 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.297340    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 20 10:18:39.297466 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.297396    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-wrzwc\""
Jul 20 10:18:39.307684 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.307666    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"]
Jul 20 10:18:39.368768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.368735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01e16418-edab-4fae-9fb2-e69d17bc8596-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.368768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.368778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e16418-edab-4fae-9fb2-e69d17bc8596-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.368990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.368819    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xqhk\" (UniqueName: \"kubernetes.io/projected/01e16418-edab-4fae-9fb2-e69d17bc8596-kube-api-access-4xqhk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.470141 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.470106    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01e16418-edab-4fae-9fb2-e69d17bc8596-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.470302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.470165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e16418-edab-4fae-9fb2-e69d17bc8596-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.470302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.470226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xqhk\" (UniqueName: \"kubernetes.io/projected/01e16418-edab-4fae-9fb2-e69d17bc8596-kube-api-access-4xqhk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.470591 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.470543    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01e16418-edab-4fae-9fb2-e69d17bc8596-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.472768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.472740    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e16418-edab-4fae-9fb2-e69d17bc8596-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.480804 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.480783    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xqhk\" (UniqueName: \"kubernetes.io/projected/01e16418-edab-4fae-9fb2-e69d17bc8596-kube-api-access-4xqhk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.599926 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.599841    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:39.741014 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.740984    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"]
Jul 20 10:18:39.743095 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:18:39.743070    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e16418_edab_4fae_9fb2_e69d17bc8596.slice/crio-aea65a95ecdf8e4f4ec1483f65392b5e628bf95bd7c246afe3dee19058e1f7f8 WatchSource:0}: Error finding container aea65a95ecdf8e4f4ec1483f65392b5e628bf95bd7c246afe3dee19058e1f7f8: Status 404 returned error can't find the container with id aea65a95ecdf8e4f4ec1483f65392b5e628bf95bd7c246afe3dee19058e1f7f8
Jul 20 10:18:39.763699 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:39.763674    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd" event={"ID":"01e16418-edab-4fae-9fb2-e69d17bc8596","Type":"ContainerStarted","Data":"aea65a95ecdf8e4f4ec1483f65392b5e628bf95bd7c246afe3dee19058e1f7f8"}
Jul 20 10:18:40.769029 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:40.768996    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd" event={"ID":"01e16418-edab-4fae-9fb2-e69d17bc8596","Type":"ContainerStarted","Data":"a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a"}
Jul 20 10:18:40.769422 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:40.769058    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:18:40.792521 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:40.792465    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd" podStartSLOduration=1.79245215 podStartE2EDuration="1.79245215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:18:40.790621154 +0000 UTC m=+1300.803723052" watchObservedRunningTime="2026-07-20 10:18:40.79245215 +0000 UTC m=+1300.805554045"
Jul 20 10:18:43.325963 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:43.325926    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"]
Jul 20 10:18:43.337847 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:43.337819    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"]
Jul 20 10:18:58.350125 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.350060    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" containerID="cri-o://3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090" gracePeriod=45
Jul 20 10:18:58.357302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.357253    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc" podUID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" containerName="main" containerID="cri-o://64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789" gracePeriod=45
Jul 20 10:18:58.532095 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.532054    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 20 10:18:58.546338 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.546302    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"]
Jul 20 10:18:58.549969 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.549948    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.552970 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.552896    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 20 10:18:58.553586 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.553523    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 20 10:18:58.553884 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.553637    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-fpvr9\""
Jul 20 10:18:58.553884 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.553572    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 20 10:18:58.554125 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.554108    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 20 10:18:58.580625 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.580593    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"]
Jul 20 10:18:58.628326 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.628301    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:18:58.635637 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.635610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpqg\" (UniqueName: \"kubernetes.io/projected/10a23ac1-c878-4201-b3de-63d83811ad0f-kube-api-access-ljpqg\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.635771 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.635651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-tmp-dir\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.635771 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.635677    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a23ac1-c878-4201-b3de-63d83811ad0f-tls-certs\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.635771 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.635751    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-home\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.635908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.635823    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-dshm\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.635908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.635851    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-model-cache\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.635908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.635872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.736755 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.736723    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tokenizer-uds\") pod \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") "
Jul 20 10:18:58.736755 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.736759    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kserve-provision-location\") pod \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") "
Jul 20 10:18:58.737018 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.736803    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tls-certs\") pod \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") "
Jul 20 10:18:58.737018 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.736866    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdjbz\" (UniqueName: \"kubernetes.io/projected/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kube-api-access-xdjbz\") pod \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\" (UID: \"b99ce9e9-0067-46b0-a74e-9b05501a3c02\") "
Jul 20 10:18:58.737018 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.736996    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-dshm\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.737182 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737032    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-model-cache\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.737182 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737055    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b99ce9e9-0067-46b0-a74e-9b05501a3c02" (UID: "b99ce9e9-0067-46b0-a74e-9b05501a3c02"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:58.737182 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.737182 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpqg\" (UniqueName: \"kubernetes.io/projected/10a23ac1-c878-4201-b3de-63d83811ad0f-kube-api-access-ljpqg\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.737182 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-tmp-dir\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.737182 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737181    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a23ac1-c878-4201-b3de-63d83811ad0f-tls-certs\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.737455 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737228    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-home\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.737455 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737328    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:58.737455 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737422    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-model-cache\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.738148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737768    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.738148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737862    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b99ce9e9-0067-46b0-a74e-9b05501a3c02" (UID: "b99ce9e9-0067-46b0-a74e-9b05501a3c02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:58.738148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.737896    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-home\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.738380 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.738217    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-tmp-dir\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.739372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.739348    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b99ce9e9-0067-46b0-a74e-9b05501a3c02" (UID: "b99ce9e9-0067-46b0-a74e-9b05501a3c02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:18:58.739692 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.739503    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kube-api-access-xdjbz" (OuterVolumeSpecName: "kube-api-access-xdjbz") pod "b99ce9e9-0067-46b0-a74e-9b05501a3c02" (UID: "b99ce9e9-0067-46b0-a74e-9b05501a3c02"). InnerVolumeSpecName "kube-api-access-xdjbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:18:58.739692 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.739624    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-dshm\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.740413 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.740392    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a23ac1-c878-4201-b3de-63d83811ad0f-tls-certs\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.750951 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.750928    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpqg\" (UniqueName: \"kubernetes.io/projected/10a23ac1-c878-4201-b3de-63d83811ad0f-kube-api-access-ljpqg\") pod \"auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.837896 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.837863    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:58.837896 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.837895    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99ce9e9-0067-46b0-a74e-9b05501a3c02-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:58.837896 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.837905    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjbz\" (UniqueName: \"kubernetes.io/projected/b99ce9e9-0067-46b0-a74e-9b05501a3c02-kube-api-access-xdjbz\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:58.850861 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.850830    2572 generic.go:356] "Generic (PLEG): container finished" podID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" containerID="64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789" exitCode=0
Jul 20 10:18:58.851036 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.850880    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"
Jul 20 10:18:58.851036 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.850891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc" event={"ID":"b99ce9e9-0067-46b0-a74e-9b05501a3c02","Type":"ContainerDied","Data":"64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789"}
Jul 20 10:18:58.851036 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.850944    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc" event={"ID":"b99ce9e9-0067-46b0-a74e-9b05501a3c02","Type":"ContainerDied","Data":"81842c5dc29d5bceca62e33ca775ec21910792077d0458375b0f752a6c6d4348"}
Jul 20 10:18:58.851036 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.850975    2572 scope.go:117] "RemoveContainer" containerID="64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789"
Jul 20 10:18:58.860539 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.860514    2572 scope.go:117] "RemoveContainer" containerID="91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da"
Jul 20 10:18:58.862811 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.862790    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:18:58.873658 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.873633    2572 scope.go:117] "RemoveContainer" containerID="64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789"
Jul 20 10:18:58.874426 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:18:58.873991    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789\": container with ID starting with 64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789 not found: ID does not exist" containerID="64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789"
Jul 20 10:18:58.874426 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.874027    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789"} err="failed to get container status \"64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789\": rpc error: code = NotFound desc = could not find container \"64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789\": container with ID starting with 64568097fca2d9c1da0e8b812fed17dd838a87ce73d19005684bf7cfa7d68789 not found: ID does not exist"
Jul 20 10:18:58.874426 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.874053    2572 scope.go:117] "RemoveContainer" containerID="91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da"
Jul 20 10:18:58.874965 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:18:58.874849    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da\": container with ID starting with 91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da not found: ID does not exist" containerID="91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da"
Jul 20 10:18:58.875295 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.875265    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da"} err="failed to get container status \"91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da\": rpc error: code = NotFound desc = could not find container \"91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da\": container with ID starting with 91f5006d7c75b8898a20f12d2451c53210309ba663126fd2bd198966fa8d07da not found: ID does not exist"
Jul 20 10:18:58.916340 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.916303    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"]
Jul 20 10:18:58.924463 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:58.923983    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-794498b458-mt7dc"]
Jul 20 10:18:59.021978 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.021955    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"]
Jul 20 10:18:59.025759 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:18:59.025734    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a23ac1_c878_4201_b3de_63d83811ad0f.slice/crio-d36493dda9f352bcad8f0be51f93745c842f541b243f2db7520e9f84c02359cc WatchSource:0}: Error finding container d36493dda9f352bcad8f0be51f93745c842f541b243f2db7520e9f84c02359cc: Status 404 returned error can't find the container with id d36493dda9f352bcad8f0be51f93745c842f541b243f2db7520e9f84c02359cc
Jul 20 10:18:59.095768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.095743    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:18:59.139828 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.139729    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tls-certs\") pod \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") "
Jul 20 10:18:59.139828 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.139792    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kserve-provision-location\") pod \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") "
Jul 20 10:18:59.139828 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.139820    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-dshm\") pod \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") "
Jul 20 10:18:59.140102 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.139863    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-home\") pod \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") "
Jul 20 10:18:59.140102 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.139892    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcmjt\" (UniqueName: \"kubernetes.io/projected/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kube-api-access-hcmjt\") pod \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") "
Jul 20 10:18:59.140102 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.139937    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tmp-dir\") pod \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") "
Jul 20 10:18:59.140102 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.139984    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-model-cache\") pod \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\" (UID: \"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862\") "
Jul 20 10:18:59.140434 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.140403    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-model-cache" (OuterVolumeSpecName: "model-cache") pod "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" (UID: "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:59.140949 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.140912    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-home" (OuterVolumeSpecName: "home") pod "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" (UID: "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:59.143375 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.143348    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" (UID: "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:18:59.143819 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.143777    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kube-api-access-hcmjt" (OuterVolumeSpecName: "kube-api-access-hcmjt") pod "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" (UID: "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862"). InnerVolumeSpecName "kube-api-access-hcmjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:18:59.143819 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.143773    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-dshm" (OuterVolumeSpecName: "dshm") pod "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" (UID: "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:59.152073 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.152045    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" (UID: "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:59.201835 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.201796    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" (UID: "55e3988c-bb7d-4c82-bb5d-f8bb06dcd862"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:59.241198 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.241163    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:59.241198 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.241195    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:59.241386 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.241209    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:59.241386 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.241220    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcmjt\" (UniqueName: \"kubernetes.io/projected/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-kube-api-access-hcmjt\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:59.241386 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.241232    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:59.241386 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.241243    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:59.241386 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.241251    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:18:59.856516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.856479    2572 generic.go:356] "Generic (PLEG): container finished" podID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerID="3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090" exitCode=0
Jul 20 10:18:59.857002 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.856539    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"
Jul 20 10:18:59.857002 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.856583    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" event={"ID":"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862","Type":"ContainerDied","Data":"3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090"}
Jul 20 10:18:59.857002 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.856642    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7" event={"ID":"55e3988c-bb7d-4c82-bb5d-f8bb06dcd862","Type":"ContainerDied","Data":"24ac124f1064b2f3e9063744cd047ca3b4bcc0770429eb4feaade4deb9776097"}
Jul 20 10:18:59.857002 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.856676    2572 scope.go:117] "RemoveContainer" containerID="3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090"
Jul 20 10:18:59.859252 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.859225    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" event={"ID":"10a23ac1-c878-4201-b3de-63d83811ad0f","Type":"ContainerStarted","Data":"df46cc627e314287b6816c0058d56b1ad82448033dbbfb38f53827f2560ef674"}
Jul 20 10:18:59.859373 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.859278    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" event={"ID":"10a23ac1-c878-4201-b3de-63d83811ad0f","Type":"ContainerStarted","Data":"d36493dda9f352bcad8f0be51f93745c842f541b243f2db7520e9f84c02359cc"}
Jul 20 10:18:59.865459 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.865440    2572 scope.go:117] "RemoveContainer" containerID="407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf"
Jul 20 10:18:59.873207 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.873192    2572 scope.go:117] "RemoveContainer" containerID="3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090"
Jul 20 10:18:59.873438 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:18:59.873422    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090\": container with ID starting with 3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090 not found: ID does not exist" containerID="3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090"
Jul 20 10:18:59.873540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.873448    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090"} err="failed to get container status \"3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090\": rpc error: code = NotFound desc = could not find container \"3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090\": container with ID starting with 3dbd39decd1dcf27f61d652d724eecaf2b3e2f565482a65b7254a8393a6ab090 not found: ID does not exist"
Jul 20 10:18:59.873540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.873473    2572 scope.go:117] "RemoveContainer" containerID="407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf"
Jul 20 10:18:59.873747 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:18:59.873732    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf\": container with ID starting with 407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf not found: ID does not exist" containerID="407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf"
Jul 20 10:18:59.873788 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.873752    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf"} err="failed to get container status \"407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf\": rpc error: code = NotFound desc = could not find container \"407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf\": container with ID starting with 407b1b1be7288044f29990209cffff8a92fdf45c165adc56360b201c0df48cdf not found: ID does not exist"
Jul 20 10:18:59.942709 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.942648    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"]
Jul 20 10:18:59.945089 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:18:59.945057    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c997987dd-r59r7"]
Jul 20 10:19:00.519690 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:00.519659    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" path="/var/lib/kubelet/pods/55e3988c-bb7d-4c82-bb5d-f8bb06dcd862/volumes"
Jul 20 10:19:00.520372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:00.520350    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" path="/var/lib/kubelet/pods/b99ce9e9-0067-46b0-a74e-9b05501a3c02/volumes"
Jul 20 10:19:02.874718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:02.874685    2572 generic.go:356] "Generic (PLEG): container finished" podID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerID="df46cc627e314287b6816c0058d56b1ad82448033dbbfb38f53827f2560ef674" exitCode=0
Jul 20 10:19:02.875102 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:02.874759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" event={"ID":"10a23ac1-c878-4201-b3de-63d83811ad0f","Type":"ContainerDied","Data":"df46cc627e314287b6816c0058d56b1ad82448033dbbfb38f53827f2560ef674"}
Jul 20 10:19:03.882154 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:03.882120    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" event={"ID":"10a23ac1-c878-4201-b3de-63d83811ad0f","Type":"ContainerStarted","Data":"d449b1b438a1ced00b2fd6d7ec6f5efba7a0e1ce9c47fbf69f9dcfd38977cc54"}
Jul 20 10:19:03.907826 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:03.907772    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podStartSLOduration=2.61088088 podStartE2EDuration="5.907755771s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.296874891s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:19:03.907619075 +0000 UTC m=+1323.920720973" watchObservedRunningTime="2026-07-20 10:19:03.907755771 +0000 UTC m=+1323.920857668"
Jul 20 10:19:04.863611 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:04.863574    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:19:08.863670 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:08.863632    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:19:08.865229 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:08.865194    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:19:11.775660 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:11.775624    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:19:18.863860 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:18.863819    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:19:28.864372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:28.864276    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:19:37.451029 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.450988    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"]
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451291    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451302    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451314    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" containerName="main"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451319    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" containerName="main"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451326    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" containerName="storage-initializer"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451333    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" containerName="storage-initializer"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451350    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="storage-initializer"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451355    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="storage-initializer"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451401    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b99ce9e9-0067-46b0-a74e-9b05501a3c02" containerName="main"
Jul 20 10:19:37.451464 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.451410    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e3988c-bb7d-4c82-bb5d-f8bb06dcd862" containerName="main"
Jul 20 10:19:37.453596 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.453576    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.457486 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.457465    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-74whd\""
Jul 20 10:19:37.457606 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.457465    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 20 10:19:37.468579 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.468536    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"]
Jul 20 10:19:37.557418 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.557379    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.557621 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.557467    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.557621 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.557510    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbxf\" (UniqueName: \"kubernetes.io/projected/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-kube-api-access-fwbxf\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.658822 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.658776    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.659008 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.658851    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbxf\" (UniqueName: \"kubernetes.io/projected/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-kube-api-access-fwbxf\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.659008 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.658919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.659336 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.659311    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.661280 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.661255    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.670479 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.670457    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbxf\" (UniqueName: \"kubernetes.io/projected/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-kube-api-access-fwbxf\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.766723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.766638    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:37.901811 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:37.901789    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"]
Jul 20 10:19:37.905030 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:19:37.905005    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a570b5_0d83_42c8_bdf5_0fca0df11cb8.slice/crio-dd650d2502760a2c96b15f05da8b5df41fadfb0733d9775b8f01ccb6a35f12d8 WatchSource:0}: Error finding container dd650d2502760a2c96b15f05da8b5df41fadfb0733d9775b8f01ccb6a35f12d8: Status 404 returned error can't find the container with id dd650d2502760a2c96b15f05da8b5df41fadfb0733d9775b8f01ccb6a35f12d8
Jul 20 10:19:38.031257 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:38.031170    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx" event={"ID":"35a570b5-0d83-42c8-bdf5-0fca0df11cb8","Type":"ContainerStarted","Data":"729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511"}
Jul 20 10:19:38.031495 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:38.031474    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx" event={"ID":"35a570b5-0d83-42c8-bdf5-0fca0df11cb8","Type":"ContainerStarted","Data":"dd650d2502760a2c96b15f05da8b5df41fadfb0733d9775b8f01ccb6a35f12d8"}
Jul 20 10:19:38.031672 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:38.031638    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:19:38.052704 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:38.052653    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx" podStartSLOduration=1.052636466 podStartE2EDuration="1.052636466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:19:38.050170449 +0000 UTC m=+1358.063272347" watchObservedRunningTime="2026-07-20 10:19:38.052636466 +0000 UTC m=+1358.065738373"
Jul 20 10:19:38.864306 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:38.864262    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:19:48.863848 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:48.863805    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:19:58.863660 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:19:58.863617    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:20:08.863686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:08.863639    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:20:09.037917 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:09.037885    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:20:10.554631 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:10.554598    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"]
Jul 20 10:20:18.864188 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:18.864137    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:20:19.038100 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:19.038060    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx" podUID="35a570b5-0d83-42c8-bdf5-0fca0df11cb8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:20:25.576922 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:25.576876    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx" podUID="35a570b5-0d83-42c8-bdf5-0fca0df11cb8" containerName="main" containerID="cri-o://729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511" gracePeriod=45
Jul 20 10:20:25.843415 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:25.843393    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:20:25.969385 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:25.969356    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tokenizer-uds\") pod \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") "
Jul 20 10:20:25.969574 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:25.969400    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwbxf\" (UniqueName: \"kubernetes.io/projected/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-kube-api-access-fwbxf\") pod \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") "
Jul 20 10:20:25.969574 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:25.969432    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tls-certs\") pod \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\" (UID: \"35a570b5-0d83-42c8-bdf5-0fca0df11cb8\") "
Jul 20 10:20:25.969700 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:25.969658    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "35a570b5-0d83-42c8-bdf5-0fca0df11cb8" (UID: "35a570b5-0d83-42c8-bdf5-0fca0df11cb8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:25.971702 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:25.971675    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-kube-api-access-fwbxf" (OuterVolumeSpecName: "kube-api-access-fwbxf") pod "35a570b5-0d83-42c8-bdf5-0fca0df11cb8" (UID: "35a570b5-0d83-42c8-bdf5-0fca0df11cb8"). InnerVolumeSpecName "kube-api-access-fwbxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:25.971702 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:25.971690    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35a570b5-0d83-42c8-bdf5-0fca0df11cb8" (UID: "35a570b5-0d83-42c8-bdf5-0fca0df11cb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:26.070532 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.070486    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:20:26.070532 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.070527    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbxf\" (UniqueName: \"kubernetes.io/projected/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-kube-api-access-fwbxf\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:20:26.070532 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.070539    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a570b5-0d83-42c8-bdf5-0fca0df11cb8-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:20:26.259665 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.259625    2572 generic.go:356] "Generic (PLEG): container finished" podID="35a570b5-0d83-42c8-bdf5-0fca0df11cb8" containerID="729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511" exitCode=0
Jul 20 10:20:26.259665 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.259663    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"
Jul 20 10:20:26.259872 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.259675    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx" event={"ID":"35a570b5-0d83-42c8-bdf5-0fca0df11cb8","Type":"ContainerDied","Data":"729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511"}
Jul 20 10:20:26.259872 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.259731    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx" event={"ID":"35a570b5-0d83-42c8-bdf5-0fca0df11cb8","Type":"ContainerDied","Data":"dd650d2502760a2c96b15f05da8b5df41fadfb0733d9775b8f01ccb6a35f12d8"}
Jul 20 10:20:26.259872 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.259763    2572 scope.go:117] "RemoveContainer" containerID="729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511"
Jul 20 10:20:26.268522 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.268502    2572 scope.go:117] "RemoveContainer" containerID="729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511"
Jul 20 10:20:26.268824 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:20:26.268800    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511\": container with ID starting with 729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511 not found: ID does not exist" containerID="729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511"
Jul 20 10:20:26.268886 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.268833    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511"} err="failed to get container status \"729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511\": rpc error: code = NotFound desc = could not find container \"729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511\": container with ID starting with 729b39547e9b7b86c343a071ec40532f4598630854bd671fbe7ebf0884f86511 not found: ID does not exist"
Jul 20 10:20:26.281195 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.281162    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"]
Jul 20 10:20:26.286646 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.286620    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7rbvx"]
Jul 20 10:20:26.519757 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:26.519679    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a570b5-0d83-42c8-bdf5-0fca0df11cb8" path="/var/lib/kubelet/pods/35a570b5-0d83-42c8-bdf5-0fca0df11cb8/volumes"
Jul 20 10:20:28.863671 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:28.863626    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:20:30.328069 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:30.328035    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"]
Jul 20 10:20:31.776378 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:31.776343    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd" podUID="01e16418-edab-4fae-9fb2-e69d17bc8596" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:20:38.864107 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:38.864060    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:20:41.776420 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:41.776374    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd" podUID="01e16418-edab-4fae-9fb2-e69d17bc8596" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:20:45.348778 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.348716    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd" podUID="01e16418-edab-4fae-9fb2-e69d17bc8596" containerName="main" containerID="cri-o://a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a" gracePeriod=45
Jul 20 10:20:45.611004 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.610937    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:20:45.734439 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.734400    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xqhk\" (UniqueName: \"kubernetes.io/projected/01e16418-edab-4fae-9fb2-e69d17bc8596-kube-api-access-4xqhk\") pod \"01e16418-edab-4fae-9fb2-e69d17bc8596\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") "
Jul 20 10:20:45.734635 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.734465    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e16418-edab-4fae-9fb2-e69d17bc8596-tls-certs\") pod \"01e16418-edab-4fae-9fb2-e69d17bc8596\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") "
Jul 20 10:20:45.734635 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.734502    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01e16418-edab-4fae-9fb2-e69d17bc8596-tokenizer-uds\") pod \"01e16418-edab-4fae-9fb2-e69d17bc8596\" (UID: \"01e16418-edab-4fae-9fb2-e69d17bc8596\") "
Jul 20 10:20:45.734858 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.734839    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e16418-edab-4fae-9fb2-e69d17bc8596-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "01e16418-edab-4fae-9fb2-e69d17bc8596" (UID: "01e16418-edab-4fae-9fb2-e69d17bc8596"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:45.736545 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.736521    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e16418-edab-4fae-9fb2-e69d17bc8596-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01e16418-edab-4fae-9fb2-e69d17bc8596" (UID: "01e16418-edab-4fae-9fb2-e69d17bc8596"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:45.736623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.736589    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e16418-edab-4fae-9fb2-e69d17bc8596-kube-api-access-4xqhk" (OuterVolumeSpecName: "kube-api-access-4xqhk") pod "01e16418-edab-4fae-9fb2-e69d17bc8596" (UID: "01e16418-edab-4fae-9fb2-e69d17bc8596"). InnerVolumeSpecName "kube-api-access-4xqhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:45.836023 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.835988    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xqhk\" (UniqueName: \"kubernetes.io/projected/01e16418-edab-4fae-9fb2-e69d17bc8596-kube-api-access-4xqhk\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:20:45.836023 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.836017    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e16418-edab-4fae-9fb2-e69d17bc8596-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:20:45.836023 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:45.836026    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01e16418-edab-4fae-9fb2-e69d17bc8596-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:20:46.354492 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.354460    2572 generic.go:356] "Generic (PLEG): container finished" podID="01e16418-edab-4fae-9fb2-e69d17bc8596" containerID="a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a" exitCode=0
Jul 20 10:20:46.354934 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.354520    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"
Jul 20 10:20:46.354934 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.354568    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd" event={"ID":"01e16418-edab-4fae-9fb2-e69d17bc8596","Type":"ContainerDied","Data":"a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a"}
Jul 20 10:20:46.354934 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.354647    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd" event={"ID":"01e16418-edab-4fae-9fb2-e69d17bc8596","Type":"ContainerDied","Data":"aea65a95ecdf8e4f4ec1483f65392b5e628bf95bd7c246afe3dee19058e1f7f8"}
Jul 20 10:20:46.354934 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.354671    2572 scope.go:117] "RemoveContainer" containerID="a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a"
Jul 20 10:20:46.363311 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.363292    2572 scope.go:117] "RemoveContainer" containerID="a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a"
Jul 20 10:20:46.363618 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:20:46.363594    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a\": container with ID starting with a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a not found: ID does not exist" containerID="a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a"
Jul 20 10:20:46.363682 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.363625    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a"} err="failed to get container status \"a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a\": rpc error: code = NotFound desc = could not find container \"a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a\": container with ID starting with a2fb3534779cfa2a8868bd1b658f7aaca65384fb6c859dfc21975b046c22fa2a not found: ID does not exist"
Jul 20 10:20:46.376473 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.376449    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"]
Jul 20 10:20:46.380302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.380278    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenq6vd"]
Jul 20 10:20:46.519264 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:46.519238    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01e16418-edab-4fae-9fb2-e69d17bc8596" path="/var/lib/kubelet/pods/01e16418-edab-4fae-9fb2-e69d17bc8596/volumes"
Jul 20 10:20:48.863448 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:48.863408    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 20 10:20:58.873696 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:58.873615    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:20:58.883058 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:58.883028    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:20:59.217865 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.217829    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"]
Jul 20 10:20:59.218160 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.218148    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e16418-edab-4fae-9fb2-e69d17bc8596" containerName="main"
Jul 20 10:20:59.218208 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.218161    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e16418-edab-4fae-9fb2-e69d17bc8596" containerName="main"
Jul 20 10:20:59.218208 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.218175    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a570b5-0d83-42c8-bdf5-0fca0df11cb8" containerName="main"
Jul 20 10:20:59.218208 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.218181    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a570b5-0d83-42c8-bdf5-0fca0df11cb8" containerName="main"
Jul 20 10:20:59.218297 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.218232    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a570b5-0d83-42c8-bdf5-0fca0df11cb8" containerName="main"
Jul 20 10:20:59.218297 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.218241    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e16418-edab-4fae-9fb2-e69d17bc8596" containerName="main"
Jul 20 10:20:59.222363 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.222344    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.226216 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.226198    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-9zwml\""
Jul 20 10:20:59.226373 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.226356    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 20 10:20:59.226550 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.226536    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 20 10:20:59.227602 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.227580    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 20 10:20:59.227706 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.227605    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-hlwqc\""
Jul 20 10:20:59.234985 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.234967    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"]
Jul 20 10:20:59.246948 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.246919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.247051 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.246957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.247100 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.247062    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splj8\" (UniqueName: \"kubernetes.io/projected/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-kube-api-access-splj8\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.347765 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.347730    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.347765 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.347764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.348007 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.347796    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splj8\" (UniqueName: \"kubernetes.io/projected/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-kube-api-access-splj8\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.348178 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.348156    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.350219 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.350190    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.368484 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.368463    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-splj8\" (UniqueName: \"kubernetes.io/projected/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-kube-api-access-splj8\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.532435 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.532350    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:20:59.666518 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:20:59.666492    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"]
Jul 20 10:20:59.668797 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:20:59.668771    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c5f6ff_fdc9_45fb_bc2a_3e24aedc09d7.slice/crio-ba1f3a0275889481e0b9217ea293850f36f8025998eb1bb366b14008bce07065 WatchSource:0}: Error finding container ba1f3a0275889481e0b9217ea293850f36f8025998eb1bb366b14008bce07065: Status 404 returned error can't find the container with id ba1f3a0275889481e0b9217ea293850f36f8025998eb1bb366b14008bce07065
Jul 20 10:21:00.416711 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:00.416676    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll" event={"ID":"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7","Type":"ContainerStarted","Data":"da2dd74a88827877c02b3aaf38234974aebdc03b567d5372418eddaf7d51641b"}
Jul 20 10:21:00.417075 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:00.416727    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll" event={"ID":"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7","Type":"ContainerStarted","Data":"ba1f3a0275889481e0b9217ea293850f36f8025998eb1bb366b14008bce07065"}
Jul 20 10:21:00.417075 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:00.416808    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:21:00.438729 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:00.438686    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll" podStartSLOduration=1.438672102 podStartE2EDuration="1.438672102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:21:00.437294915 +0000 UTC m=+1440.450396815" watchObservedRunningTime="2026-07-20 10:21:00.438672102 +0000 UTC m=+1440.451773998"
Jul 20 10:21:11.934228 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:11.934190    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"]
Jul 20 10:21:26.957151 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:26.957105    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main" containerID="cri-o://d449b1b438a1ced00b2fd6d7ec6f5efba7a0e1ce9c47fbf69f9dcfd38977cc54" gracePeriod=45
Jul 20 10:21:27.532604 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.532540    2572 generic.go:356] "Generic (PLEG): container finished" podID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerID="d449b1b438a1ced00b2fd6d7ec6f5efba7a0e1ce9c47fbf69f9dcfd38977cc54" exitCode=0
Jul 20 10:21:27.532604 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.532586    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" event={"ID":"10a23ac1-c878-4201-b3de-63d83811ad0f","Type":"ContainerDied","Data":"d449b1b438a1ced00b2fd6d7ec6f5efba7a0e1ce9c47fbf69f9dcfd38977cc54"}
Jul 20 10:21:27.633012 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.632989    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:21:27.795834 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.795749    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a23ac1-c878-4201-b3de-63d83811ad0f-tls-certs\") pod \"10a23ac1-c878-4201-b3de-63d83811ad0f\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") "
Jul 20 10:21:27.795834 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.795795    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-home\") pod \"10a23ac1-c878-4201-b3de-63d83811ad0f\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") "
Jul 20 10:21:27.796052 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.795841    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljpqg\" (UniqueName: \"kubernetes.io/projected/10a23ac1-c878-4201-b3de-63d83811ad0f-kube-api-access-ljpqg\") pod \"10a23ac1-c878-4201-b3de-63d83811ad0f\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") "
Jul 20 10:21:27.796052 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.795856    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-dshm\") pod \"10a23ac1-c878-4201-b3de-63d83811ad0f\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") "
Jul 20 10:21:27.796052 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.795872    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-model-cache\") pod \"10a23ac1-c878-4201-b3de-63d83811ad0f\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") "
Jul 20 10:21:27.796052 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.795892    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-tmp-dir\") pod \"10a23ac1-c878-4201-b3de-63d83811ad0f\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") "
Jul 20 10:21:27.796052 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.795914    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-kserve-provision-location\") pod \"10a23ac1-c878-4201-b3de-63d83811ad0f\" (UID: \"10a23ac1-c878-4201-b3de-63d83811ad0f\") "
Jul 20 10:21:27.796314 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.796168    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "10a23ac1-c878-4201-b3de-63d83811ad0f" (UID: "10a23ac1-c878-4201-b3de-63d83811ad0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:21:27.796516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.796495    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-home" (OuterVolumeSpecName: "home") pod "10a23ac1-c878-4201-b3de-63d83811ad0f" (UID: "10a23ac1-c878-4201-b3de-63d83811ad0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:21:27.798127 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.798096    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a23ac1-c878-4201-b3de-63d83811ad0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10a23ac1-c878-4201-b3de-63d83811ad0f" (UID: "10a23ac1-c878-4201-b3de-63d83811ad0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:21:27.798485 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.798463    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a23ac1-c878-4201-b3de-63d83811ad0f-kube-api-access-ljpqg" (OuterVolumeSpecName: "kube-api-access-ljpqg") pod "10a23ac1-c878-4201-b3de-63d83811ad0f" (UID: "10a23ac1-c878-4201-b3de-63d83811ad0f"). InnerVolumeSpecName "kube-api-access-ljpqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:21:27.798571 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.798537    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-dshm" (OuterVolumeSpecName: "dshm") pod "10a23ac1-c878-4201-b3de-63d83811ad0f" (UID: "10a23ac1-c878-4201-b3de-63d83811ad0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:21:27.813257 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.813226    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10a23ac1-c878-4201-b3de-63d83811ad0f" (UID: "10a23ac1-c878-4201-b3de-63d83811ad0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:21:27.864213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.864155    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10a23ac1-c878-4201-b3de-63d83811ad0f" (UID: "10a23ac1-c878-4201-b3de-63d83811ad0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:21:27.897293 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.897265    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a23ac1-c878-4201-b3de-63d83811ad0f-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.897293 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.897292    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.897394 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.897302    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljpqg\" (UniqueName: \"kubernetes.io/projected/10a23ac1-c878-4201-b3de-63d83811ad0f-kube-api-access-ljpqg\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.897394 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.897311    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.897394 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.897320    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.897394 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.897329    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:21:27.897394 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:27.897336    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10a23ac1-c878-4201-b3de-63d83811ad0f-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:21:28.537708 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:28.537680    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj" event={"ID":"10a23ac1-c878-4201-b3de-63d83811ad0f","Type":"ContainerDied","Data":"d36493dda9f352bcad8f0be51f93745c842f541b243f2db7520e9f84c02359cc"}
Jul 20 10:21:28.538051 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:28.537719    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"
Jul 20 10:21:28.538051 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:28.537738    2572 scope.go:117] "RemoveContainer" containerID="d449b1b438a1ced00b2fd6d7ec6f5efba7a0e1ce9c47fbf69f9dcfd38977cc54"
Jul 20 10:21:28.545787 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:28.545768    2572 scope.go:117] "RemoveContainer" containerID="df46cc627e314287b6816c0058d56b1ad82448033dbbfb38f53827f2560ef674"
Jul 20 10:21:28.558084 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:28.558058    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"]
Jul 20 10:21:28.562043 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:28.562023    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d9bb8d8b9-kdklj"]
Jul 20 10:21:30.519015 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:30.518985    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" path="/var/lib/kubelet/pods/10a23ac1-c878-4201-b3de-63d83811ad0f/volumes"
Jul 20 10:21:31.424195 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:31.424166    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:21:44.667243 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.667204    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"]
Jul 20 10:21:44.667682 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.667495    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main"
Jul 20 10:21:44.667682 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.667505    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main"
Jul 20 10:21:44.667682 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.667515    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="storage-initializer"
Jul 20 10:21:44.667682 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.667521    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="storage-initializer"
Jul 20 10:21:44.667682 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.667595    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a23ac1-c878-4201-b3de-63d83811ad0f" containerName="main"
Jul 20 10:21:44.670723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.670705    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.673486 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.673467    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-g2mnb\""
Jul 20 10:21:44.674790 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.674774    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 20 10:21:44.696003 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.695977    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"]
Jul 20 10:21:44.726253 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.726220    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlx76\" (UniqueName: \"kubernetes.io/projected/fe54f5c7-d36e-4016-b0c3-2cc93c610612-kube-api-access-qlx76\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.726253 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.726255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.726455 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.726285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.826885 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.826845    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.827063 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.826958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlx76\" (UniqueName: \"kubernetes.io/projected/fe54f5c7-d36e-4016-b0c3-2cc93c610612-kube-api-access-qlx76\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.827063 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.826990    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.827357 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.827337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.829276 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.829250    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.837971 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.837949    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlx76\" (UniqueName: \"kubernetes.io/projected/fe54f5c7-d36e-4016-b0c3-2cc93c610612-kube-api-access-qlx76\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:44.980525 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:44.980489    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:45.125364 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:45.125343    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"]
Jul 20 10:21:45.127831 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:21:45.127805    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe54f5c7_d36e_4016_b0c3_2cc93c610612.slice/crio-2828499d0dcdb074a28aea6893cead1b9471329935b219db944e6c46b73bebc4 WatchSource:0}: Error finding container 2828499d0dcdb074a28aea6893cead1b9471329935b219db944e6c46b73bebc4: Status 404 returned error can't find the container with id 2828499d0dcdb074a28aea6893cead1b9471329935b219db944e6c46b73bebc4
Jul 20 10:21:45.612292 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:45.612255    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8" event={"ID":"fe54f5c7-d36e-4016-b0c3-2cc93c610612","Type":"ContainerStarted","Data":"d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9"}
Jul 20 10:21:45.612470 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:45.612311    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8" event={"ID":"fe54f5c7-d36e-4016-b0c3-2cc93c610612","Type":"ContainerStarted","Data":"2828499d0dcdb074a28aea6893cead1b9471329935b219db944e6c46b73bebc4"}
Jul 20 10:21:45.612470 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:45.612349    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:21:45.633841 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:45.633783    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8" podStartSLOduration=1.6337630829999998 podStartE2EDuration="1.633763083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:21:45.631622766 +0000 UTC m=+1485.644724663" watchObservedRunningTime="2026-07-20 10:21:45.633763083 +0000 UTC m=+1485.646864980"
Jul 20 10:21:50.299325 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.299288    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"]
Jul 20 10:21:50.303545 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.303527    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.306358 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.306326    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-m44gb\""
Jul 20 10:21:50.306526 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.306433    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 20 10:21:50.306526 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.306457    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 20 10:21:50.306526 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.306489    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 20 10:21:50.306747 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.306586    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 20 10:21:50.315042 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.315022    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"]
Jul 20 10:21:50.371806 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.371768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tls-certs\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.371990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.371840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tmp-dir\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.371990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.371872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-dshm\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.371990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.371903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-home\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.371990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.371957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-model-cache\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.372163 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.372076    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kserve-provision-location\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.372163 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.372107    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcmz\" (UniqueName: \"kubernetes.io/projected/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kube-api-access-5jcmz\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.472529 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472498    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tmp-dir\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.472529 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472531    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-dshm\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.472785 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472574    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-home\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.472785 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472696    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-model-cache\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.472908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kserve-provision-location\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.472908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472832    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcmz\" (UniqueName: \"kubernetes.io/projected/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kube-api-access-5jcmz\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.472908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472881    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tls-certs\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.473073 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472940    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-home\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.473073 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.472989    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tmp-dir\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.473073 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.473026    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-model-cache\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.473197 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.473175    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kserve-provision-location\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.474843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.474823    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-dshm\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.475179 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.475163    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tls-certs\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.483007 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.482986    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcmz\" (UniqueName: \"kubernetes.io/projected/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kube-api-access-5jcmz\") pod \"auth-disabled-test-kserve-65695595d-cgcgx\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.615331 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.615243    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:21:50.787968 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:50.787937    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"]
Jul 20 10:21:50.791167 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:21:50.791143    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb3e14b_e031_4bf9_ab77_1f551e501a89.slice/crio-e23a1065a43e813c0dff4955fe3fc5df5f0c77fd0dcf95e58c5790909d5b3d7b WatchSource:0}: Error finding container e23a1065a43e813c0dff4955fe3fc5df5f0c77fd0dcf95e58c5790909d5b3d7b: Status 404 returned error can't find the container with id e23a1065a43e813c0dff4955fe3fc5df5f0c77fd0dcf95e58c5790909d5b3d7b
Jul 20 10:21:51.645597 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:51.645540    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" event={"ID":"5bb3e14b-e031-4bf9-ab77-1f551e501a89","Type":"ContainerStarted","Data":"48672dbb550fa8487362cebdfbbc294d6fedc1372359a6d9a2205429f5aff21c"}
Jul 20 10:21:51.646015 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:51.645614    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" event={"ID":"5bb3e14b-e031-4bf9-ab77-1f551e501a89","Type":"ContainerStarted","Data":"e23a1065a43e813c0dff4955fe3fc5df5f0c77fd0dcf95e58c5790909d5b3d7b"}
Jul 20 10:21:54.658521 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:54.658483    2572 generic.go:356] "Generic (PLEG): container finished" podID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerID="48672dbb550fa8487362cebdfbbc294d6fedc1372359a6d9a2205429f5aff21c" exitCode=0
Jul 20 10:21:54.658962 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:54.658574    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" event={"ID":"5bb3e14b-e031-4bf9-ab77-1f551e501a89","Type":"ContainerDied","Data":"48672dbb550fa8487362cebdfbbc294d6fedc1372359a6d9a2205429f5aff21c"}
Jul 20 10:21:55.664466 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:55.664432    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" event={"ID":"5bb3e14b-e031-4bf9-ab77-1f551e501a89","Type":"ContainerStarted","Data":"52a8db22990ba88a860cad1d77b596882b5c6c61e9bff75a544145a6a3e2a7f6"}
Jul 20 10:21:55.688494 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:55.688444    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podStartSLOduration=1.991227442 podStartE2EDuration="5.688430368s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.697202926s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:21:55.685924833 +0000 UTC m=+1495.699026728" watchObservedRunningTime="2026-07-20 10:21:55.688430368 +0000 UTC m=+1495.701532264"
Jul 20 10:21:56.615878 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:21:56.615843    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:22:00.473843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:00.473809    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:22:00.481147 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:00.481128    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:22:00.615668 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:00.615631    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:22:00.617220 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:00.617190    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:22:10.616377 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:10.616333    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:22:16.618052 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:16.618026    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:22:20.615956 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:20.615911    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:22:23.927884 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:23.927791    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"]
Jul 20 10:22:26.618432 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:26.618386    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8" podUID="fe54f5c7-d36e-4016-b0c3-2cc93c610612" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:22:30.616339 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:30.616293    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:22:36.618779 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:36.618742    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8" podUID="fe54f5c7-d36e-4016-b0c3-2cc93c610612" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:22:38.954996 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:38.954908    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8" podUID="fe54f5c7-d36e-4016-b0c3-2cc93c610612" containerName="main" containerID="cri-o://d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9" gracePeriod=45
Jul 20 10:22:39.200279 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.200255    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:22:39.289304 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.289220    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tokenizer-uds\") pod \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") "
Jul 20 10:22:39.289304 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.289270    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlx76\" (UniqueName: \"kubernetes.io/projected/fe54f5c7-d36e-4016-b0c3-2cc93c610612-kube-api-access-qlx76\") pod \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") "
Jul 20 10:22:39.289519 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.289316    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tls-certs\") pod \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\" (UID: \"fe54f5c7-d36e-4016-b0c3-2cc93c610612\") "
Jul 20 10:22:39.289519 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.289485    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "fe54f5c7-d36e-4016-b0c3-2cc93c610612" (UID: "fe54f5c7-d36e-4016-b0c3-2cc93c610612"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:39.291676 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.291647    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe54f5c7-d36e-4016-b0c3-2cc93c610612-kube-api-access-qlx76" (OuterVolumeSpecName: "kube-api-access-qlx76") pod "fe54f5c7-d36e-4016-b0c3-2cc93c610612" (UID: "fe54f5c7-d36e-4016-b0c3-2cc93c610612"). InnerVolumeSpecName "kube-api-access-qlx76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:22:39.291786 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.291656    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe54f5c7-d36e-4016-b0c3-2cc93c610612" (UID: "fe54f5c7-d36e-4016-b0c3-2cc93c610612"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:22:39.390523 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.390487    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:22:39.390523 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.390516    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlx76\" (UniqueName: \"kubernetes.io/projected/fe54f5c7-d36e-4016-b0c3-2cc93c610612-kube-api-access-qlx76\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:22:39.390523 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.390526    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe54f5c7-d36e-4016-b0c3-2cc93c610612-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:22:39.861534 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.861500    2572 generic.go:356] "Generic (PLEG): container finished" podID="fe54f5c7-d36e-4016-b0c3-2cc93c610612" containerID="d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9" exitCode=0
Jul 20 10:22:39.861754 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.861567    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"
Jul 20 10:22:39.861754 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.861580    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8" event={"ID":"fe54f5c7-d36e-4016-b0c3-2cc93c610612","Type":"ContainerDied","Data":"d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9"}
Jul 20 10:22:39.861754 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.861636    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8" event={"ID":"fe54f5c7-d36e-4016-b0c3-2cc93c610612","Type":"ContainerDied","Data":"2828499d0dcdb074a28aea6893cead1b9471329935b219db944e6c46b73bebc4"}
Jul 20 10:22:39.861754 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.861661    2572 scope.go:117] "RemoveContainer" containerID="d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9"
Jul 20 10:22:39.870786 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.870769    2572 scope.go:117] "RemoveContainer" containerID="d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9"
Jul 20 10:22:39.871038 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:22:39.871019    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9\": container with ID starting with d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9 not found: ID does not exist" containerID="d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9"
Jul 20 10:22:39.871094 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.871045    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9"} err="failed to get container status \"d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9\": rpc error: code = NotFound desc = could not find container \"d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9\": container with ID starting with d3f4a8bfc8555c49beb56276592a2d0a1cd4f334c931e5c72cfa81b430eb8da9 not found: ID does not exist"
Jul 20 10:22:39.884434 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.884402    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"]
Jul 20 10:22:39.886286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:39.886263    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69mx8"]
Jul 20 10:22:40.519797 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:40.519758    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe54f5c7-d36e-4016-b0c3-2cc93c610612" path="/var/lib/kubelet/pods/fe54f5c7-d36e-4016-b0c3-2cc93c610612/volumes"
Jul 20 10:22:40.616341 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:40.616303    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:22:43.773527 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:43.773496    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"]
Jul 20 10:22:50.616243 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:50.616191    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:22:51.423410 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:51.423371    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll" podUID="d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:22:58.794788 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:58.794732    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll" podUID="d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" containerName="main" containerID="cri-o://da2dd74a88827877c02b3aaf38234974aebdc03b567d5372418eddaf7d51641b" gracePeriod=45
Jul 20 10:22:58.952643 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:58.952600    2572 generic.go:356] "Generic (PLEG): container finished" podID="d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" containerID="da2dd74a88827877c02b3aaf38234974aebdc03b567d5372418eddaf7d51641b" exitCode=0
Jul 20 10:22:58.952835 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:58.952667    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll" event={"ID":"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7","Type":"ContainerDied","Data":"da2dd74a88827877c02b3aaf38234974aebdc03b567d5372418eddaf7d51641b"}
Jul 20 10:22:59.041581 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.041544    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:22:59.140109 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.140011    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-splj8\" (UniqueName: \"kubernetes.io/projected/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-kube-api-access-splj8\") pod \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") "
Jul 20 10:22:59.140109 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.140054    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tokenizer-uds\") pod \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") "
Jul 20 10:22:59.140109 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.140088    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tls-certs\") pod \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\" (UID: \"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7\") "
Jul 20 10:22:59.140399 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.140337    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" (UID: "d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:59.142200 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.142163    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-kube-api-access-splj8" (OuterVolumeSpecName: "kube-api-access-splj8") pod "d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" (UID: "d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7"). InnerVolumeSpecName "kube-api-access-splj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:22:59.142294 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.142169    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" (UID: "d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:22:59.240908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.240862    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splj8\" (UniqueName: \"kubernetes.io/projected/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-kube-api-access-splj8\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:22:59.240908 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.240907    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:22:59.241101 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.240924    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:22:59.957777 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.957747    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"
Jul 20 10:22:59.957777 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.957754    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll" event={"ID":"d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7","Type":"ContainerDied","Data":"ba1f3a0275889481e0b9217ea293850f36f8025998eb1bb366b14008bce07065"}
Jul 20 10:22:59.958302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.957826    2572 scope.go:117] "RemoveContainer" containerID="da2dd74a88827877c02b3aaf38234974aebdc03b567d5372418eddaf7d51641b"
Jul 20 10:22:59.981387 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.981361    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"]
Jul 20 10:22:59.986402 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:22:59.986377    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5r9ll"]
Jul 20 10:23:00.523631 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:00.523403    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" path="/var/lib/kubelet/pods/d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7/volumes"
Jul 20 10:23:00.616459 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:00.616402    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:23:10.615939 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:10.615900    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:23:13.508611 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.508570    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"]
Jul 20 10:23:13.509025 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.509009    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe54f5c7-d36e-4016-b0c3-2cc93c610612" containerName="main"
Jul 20 10:23:13.509074 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.509029    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe54f5c7-d36e-4016-b0c3-2cc93c610612" containerName="main"
Jul 20 10:23:13.509074 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.509042    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" containerName="main"
Jul 20 10:23:13.509074 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.509051    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" containerName="main"
Jul 20 10:23:13.509190 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.509136    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c5f6ff-fdc9-45fb-bc2a-3e24aedc09d7" containerName="main"
Jul 20 10:23:13.509190 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.509147    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe54f5c7-d36e-4016-b0c3-2cc93c610612" containerName="main"
Jul 20 10:23:13.512331 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.512312    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.515280 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.515256    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-vzdz8\""
Jul 20 10:23:13.515280 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.515273    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 20 10:23:13.515491 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.515387    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 20 10:23:13.515577 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.515531    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 20 10:23:13.515719 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.515686    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 20 10:23:13.515799 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.515754    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:23:13.524450 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.524424    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"]
Jul 20 10:23:13.667735 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.667693    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.667735 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.667731    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.668026 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.667759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.668026 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.667783    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwbvm\" (UniqueName: \"kubernetes.io/projected/fabba8d3-d52d-4411-bebd-895bc6e208b0-kube-api-access-nwbvm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.668026 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.667825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.668026 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.667875    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabba8d3-d52d-4411-bebd-895bc6e208b0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.668026 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.667908    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.668026 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.667980    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fabba8d3-d52d-4411-bebd-895bc6e208b0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769365 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769365 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769306    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fabba8d3-d52d-4411-bebd-895bc6e208b0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769365 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769332    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769703 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769420    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769703 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769529    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769703 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwbvm\" (UniqueName: \"kubernetes.io/projected/fabba8d3-d52d-4411-bebd-895bc6e208b0-kube-api-access-nwbvm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769703 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769631    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769703 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabba8d3-d52d-4411-bebd-895bc6e208b0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769994 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769794    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769994 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769844    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.769994 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.769948    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.770158 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.770121    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fabba8d3-d52d-4411-bebd-895bc6e208b0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.770221 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.770189    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.772079 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.772057    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.772409 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.772378    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabba8d3-d52d-4411-bebd-895bc6e208b0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.777806 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.777780    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwbvm\" (UniqueName: \"kubernetes.io/projected/fabba8d3-d52d-4411-bebd-895bc6e208b0-kube-api-access-nwbvm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.818117 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.818088    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"]
Jul 20 10:23:13.820954 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.820936    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:13.823715 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.823692    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:13.823817 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.823774    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-j4cn6\""
Jul 20 10:23:13.833530 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.833507    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"]
Jul 20 10:23:13.963148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.962997    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"]
Jul 20 10:23:13.966004 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:23:13.965969    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabba8d3_d52d_4411_bebd_895bc6e208b0.slice/crio-17ffa2d3f60e96b420bec45bbc153823a891b6e322b3d87f64207e3e9bb85499 WatchSource:0}: Error finding container 17ffa2d3f60e96b420bec45bbc153823a891b6e322b3d87f64207e3e9bb85499: Status 404 returned error can't find the container with id 17ffa2d3f60e96b420bec45bbc153823a891b6e322b3d87f64207e3e9bb85499
Jul 20 10:23:13.971700 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.971662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:13.971843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.971708    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5f5j\" (UniqueName: \"kubernetes.io/projected/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kube-api-access-z5f5j\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:13.971843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.971808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:13.971843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:13.971835    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.017011 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.016930    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" event={"ID":"fabba8d3-d52d-4411-bebd-895bc6e208b0","Type":"ContainerStarted","Data":"17ffa2d3f60e96b420bec45bbc153823a891b6e322b3d87f64207e3e9bb85499"}
Jul 20 10:23:14.073439 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.073339    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.073439 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.073400    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.073697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.073446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.073697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.073470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5f5j\" (UniqueName: \"kubernetes.io/projected/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kube-api-access-z5f5j\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.073865 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.073825    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.074005 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.073867    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.075905 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.075882    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.082450 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.082429    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5f5j\" (UniqueName: \"kubernetes.io/projected/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kube-api-access-z5f5j\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.132547 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.132503    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:14.286362 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:14.286323    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"]
Jul 20 10:23:14.290662 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:23:14.290631    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b57df06_29a1_46f2_b8a5_e162cf21f6b7.slice/crio-9e80c1fc1879c810c33252e0027efbe2f8c780f232ba9caedf790fb33981d1ff WatchSource:0}: Error finding container 9e80c1fc1879c810c33252e0027efbe2f8c780f232ba9caedf790fb33981d1ff: Status 404 returned error can't find the container with id 9e80c1fc1879c810c33252e0027efbe2f8c780f232ba9caedf790fb33981d1ff
Jul 20 10:23:15.024476 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:15.024437    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" event={"ID":"1b57df06-29a1-46f2-b8a5-e162cf21f6b7","Type":"ContainerStarted","Data":"8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d"}
Jul 20 10:23:15.025137 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:15.024500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" event={"ID":"1b57df06-29a1-46f2-b8a5-e162cf21f6b7","Type":"ContainerStarted","Data":"9e80c1fc1879c810c33252e0027efbe2f8c780f232ba9caedf790fb33981d1ff"}
Jul 20 10:23:15.026752 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:15.026722    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" event={"ID":"fabba8d3-d52d-4411-bebd-895bc6e208b0","Type":"ContainerStarted","Data":"093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c"}
Jul 20 10:23:16.038947 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:16.038900    2572 generic.go:356] "Generic (PLEG): container finished" podID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerID="8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d" exitCode=0
Jul 20 10:23:16.039367 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:16.038986    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" event={"ID":"1b57df06-29a1-46f2-b8a5-e162cf21f6b7","Type":"ContainerDied","Data":"8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d"}
Jul 20 10:23:17.047544 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:17.047497    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" event={"ID":"1b57df06-29a1-46f2-b8a5-e162cf21f6b7","Type":"ContainerStarted","Data":"dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4"}
Jul 20 10:23:17.047998 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:17.047732    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:17.075696 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:17.075619    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" podStartSLOduration=3.325120596 podStartE2EDuration="4.075602869s" totalImagesPullingTime="0s" totalInitContainerRuntime="750.482273ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:23:17.072926888 +0000 UTC m=+1577.086028795" watchObservedRunningTime="2026-07-20 10:23:17.075602869 +0000 UTC m=+1577.088704766"
Jul 20 10:23:18.052836 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:18.052801    2572 generic.go:356] "Generic (PLEG): container finished" podID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerID="093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c" exitCode=0
Jul 20 10:23:18.053221 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:18.052886    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" event={"ID":"fabba8d3-d52d-4411-bebd-895bc6e208b0","Type":"ContainerDied","Data":"093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c"}
Jul 20 10:23:19.060855 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:19.060821    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" event={"ID":"fabba8d3-d52d-4411-bebd-895bc6e208b0","Type":"ContainerStarted","Data":"d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d"}
Jul 20 10:23:19.087326 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:19.087257    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podStartSLOduration=2.578723105 podStartE2EDuration="6.087234521s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.508511416s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:23:19.084522551 +0000 UTC m=+1579.097624448" watchObservedRunningTime="2026-07-20 10:23:19.087234521 +0000 UTC m=+1579.100336419"
Jul 20 10:23:19.824391 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:19.824359    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:20.616741 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:20.616693    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:23:23.824218 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:23.824176    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:23:23.825587 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:23.825538    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 20 10:23:30.616752 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:30.616626    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:23:33.825120 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:33.825070    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 20 10:23:40.615678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:40.615628    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 20 10:23:43.824299 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:43.824246    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 20 10:23:48.055496 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:48.055466    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:23:50.626096 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:50.626067    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:23:50.634098 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:50.634070    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:23:53.824265 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:53.824151    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 20 10:23:55.615363 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:23:55.615325    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"]
Jul 20 10:24:00.559656 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:00.559627    2572 scope.go:117] "RemoveContainer" containerID="aedadc97257fc6422c11ebb02d06f5c9cd1963d927d7e8c3b4dbf157cc385b93"
Jul 20 10:24:03.824658 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:03.824612    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 20 10:24:10.638254 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:10.638208    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main" containerID="cri-o://52a8db22990ba88a860cad1d77b596882b5c6c61e9bff75a544145a6a3e2a7f6" gracePeriod=45
Jul 20 10:24:11.328340 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.328308    2572 generic.go:356] "Generic (PLEG): container finished" podID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerID="52a8db22990ba88a860cad1d77b596882b5c6c61e9bff75a544145a6a3e2a7f6" exitCode=0
Jul 20 10:24:11.328508 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.328400    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" event={"ID":"5bb3e14b-e031-4bf9-ab77-1f551e501a89","Type":"ContainerDied","Data":"52a8db22990ba88a860cad1d77b596882b5c6c61e9bff75a544145a6a3e2a7f6"}
Jul 20 10:24:11.393201 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.393168    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:24:11.506586 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.506517    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kserve-provision-location\") pod \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") "
Jul 20 10:24:11.506586 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.506592    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-dshm\") pod \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") "
Jul 20 10:24:11.506854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.506627    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tls-certs\") pod \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") "
Jul 20 10:24:11.506854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.506666    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-home\") pod \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") "
Jul 20 10:24:11.506854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.506686    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-model-cache\") pod \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") "
Jul 20 10:24:11.506854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.506739    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tmp-dir\") pod \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") "
Jul 20 10:24:11.506854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.506799    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jcmz\" (UniqueName: \"kubernetes.io/projected/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kube-api-access-5jcmz\") pod \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\" (UID: \"5bb3e14b-e031-4bf9-ab77-1f551e501a89\") "
Jul 20 10:24:11.507103 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.506959    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bb3e14b-e031-4bf9-ab77-1f551e501a89" (UID: "5bb3e14b-e031-4bf9-ab77-1f551e501a89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:11.507103 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.507039    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.507407 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.507381    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-home" (OuterVolumeSpecName: "home") pod "5bb3e14b-e031-4bf9-ab77-1f551e501a89" (UID: "5bb3e14b-e031-4bf9-ab77-1f551e501a89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:11.509121 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.509082    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bb3e14b-e031-4bf9-ab77-1f551e501a89" (UID: "5bb3e14b-e031-4bf9-ab77-1f551e501a89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:24:11.509341 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.509313    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-dshm" (OuterVolumeSpecName: "dshm") pod "5bb3e14b-e031-4bf9-ab77-1f551e501a89" (UID: "5bb3e14b-e031-4bf9-ab77-1f551e501a89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:11.509341 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.509329    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kube-api-access-5jcmz" (OuterVolumeSpecName: "kube-api-access-5jcmz") pod "5bb3e14b-e031-4bf9-ab77-1f551e501a89" (UID: "5bb3e14b-e031-4bf9-ab77-1f551e501a89"). InnerVolumeSpecName "kube-api-access-5jcmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:24:11.518890 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.518855    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bb3e14b-e031-4bf9-ab77-1f551e501a89" (UID: "5bb3e14b-e031-4bf9-ab77-1f551e501a89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:11.570165 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.570093    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bb3e14b-e031-4bf9-ab77-1f551e501a89" (UID: "5bb3e14b-e031-4bf9-ab77-1f551e501a89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:11.608318 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.608284    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.608516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.608330    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.608516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.608342    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jcmz\" (UniqueName: \"kubernetes.io/projected/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kube-api-access-5jcmz\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.608516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.608351    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.608516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.608359    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3e14b-e031-4bf9-ab77-1f551e501a89-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:24:11.608516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:11.608367    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3e14b-e031-4bf9-ab77-1f551e501a89-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:24:12.334113 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:12.334084    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"
Jul 20 10:24:12.334623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:12.334083    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx" event={"ID":"5bb3e14b-e031-4bf9-ab77-1f551e501a89","Type":"ContainerDied","Data":"e23a1065a43e813c0dff4955fe3fc5df5f0c77fd0dcf95e58c5790909d5b3d7b"}
Jul 20 10:24:12.334623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:12.334214    2572 scope.go:117] "RemoveContainer" containerID="52a8db22990ba88a860cad1d77b596882b5c6c61e9bff75a544145a6a3e2a7f6"
Jul 20 10:24:12.343444 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:12.343428    2572 scope.go:117] "RemoveContainer" containerID="48672dbb550fa8487362cebdfbbc294d6fedc1372359a6d9a2205429f5aff21c"
Jul 20 10:24:12.361577 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:12.361531    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"]
Jul 20 10:24:12.364874 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:12.364850    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65695595d-cgcgx"]
Jul 20 10:24:12.521370 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:12.521336    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" path="/var/lib/kubelet/pods/5bb3e14b-e031-4bf9-ab77-1f551e501a89/volumes"
Jul 20 10:24:13.824917 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:13.824871    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 20 10:24:23.824937 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:23.824884    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 20 10:24:33.834310 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:33.834278    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:24:33.842593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:24:33.842571    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:25:08.942767 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:08.942726    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"]
Jul 20 10:25:08.945835 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:08.945810    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"]
Jul 20 10:25:18.054613 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:18.054577    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" podUID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:25:23.964085 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:23.963979    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" podUID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerName="main" containerID="cri-o://dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4" gracePeriod=45
Jul 20 10:25:23.974537 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:23.974492    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main" containerID="cri-o://d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d" gracePeriod=45
Jul 20 10:25:24.212999 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.212974    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:25:24.320036 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.319951    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tls-certs\") pod \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") "
Jul 20 10:25:24.320036 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.320026    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kserve-provision-location\") pod \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") "
Jul 20 10:25:24.320252 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.320050    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5f5j\" (UniqueName: \"kubernetes.io/projected/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kube-api-access-z5f5j\") pod \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") "
Jul 20 10:25:24.320252 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.320075    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tokenizer-uds\") pod \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\" (UID: \"1b57df06-29a1-46f2-b8a5-e162cf21f6b7\") "
Jul 20 10:25:24.320423 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.320398    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1b57df06-29a1-46f2-b8a5-e162cf21f6b7" (UID: "1b57df06-29a1-46f2-b8a5-e162cf21f6b7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:25:24.320780 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.320751    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b57df06-29a1-46f2-b8a5-e162cf21f6b7" (UID: "1b57df06-29a1-46f2-b8a5-e162cf21f6b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:25:24.322313 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.322286    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kube-api-access-z5f5j" (OuterVolumeSpecName: "kube-api-access-z5f5j") pod "1b57df06-29a1-46f2-b8a5-e162cf21f6b7" (UID: "1b57df06-29a1-46f2-b8a5-e162cf21f6b7"). InnerVolumeSpecName "kube-api-access-z5f5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:25:24.322395 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.322323    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b57df06-29a1-46f2-b8a5-e162cf21f6b7" (UID: "1b57df06-29a1-46f2-b8a5-e162cf21f6b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:25:24.421599 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.421544    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.421599 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.421594    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5f5j\" (UniqueName: \"kubernetes.io/projected/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-kube-api-access-z5f5j\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.421599 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.421607    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.421844 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.421639    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b57df06-29a1-46f2-b8a5-e162cf21f6b7-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.618249 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.618223    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:25:24.646831 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.646800    2572 generic.go:356] "Generic (PLEG): container finished" podID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerID="dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4" exitCode=0
Jul 20 10:25:24.646994 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.646858    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"
Jul 20 10:25:24.646994 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.646902    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" event={"ID":"1b57df06-29a1-46f2-b8a5-e162cf21f6b7","Type":"ContainerDied","Data":"dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4"}
Jul 20 10:25:24.646994 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.646982    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v" event={"ID":"1b57df06-29a1-46f2-b8a5-e162cf21f6b7","Type":"ContainerDied","Data":"9e80c1fc1879c810c33252e0027efbe2f8c780f232ba9caedf790fb33981d1ff"}
Jul 20 10:25:24.647157 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.647014    2572 scope.go:117] "RemoveContainer" containerID="dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4"
Jul 20 10:25:24.649740 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.649715    2572 generic.go:356] "Generic (PLEG): container finished" podID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerID="d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d" exitCode=0
Jul 20 10:25:24.649831 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.649761    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" event={"ID":"fabba8d3-d52d-4411-bebd-895bc6e208b0","Type":"ContainerDied","Data":"d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d"}
Jul 20 10:25:24.649831 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.649802    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f" event={"ID":"fabba8d3-d52d-4411-bebd-895bc6e208b0","Type":"ContainerDied","Data":"17ffa2d3f60e96b420bec45bbc153823a891b6e322b3d87f64207e3e9bb85499"}
Jul 20 10:25:24.649973 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.649959    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"
Jul 20 10:25:24.656847 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.656828    2572 scope.go:117] "RemoveContainer" containerID="8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d"
Jul 20 10:25:24.664473 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.664450    2572 scope.go:117] "RemoveContainer" containerID="dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4"
Jul 20 10:25:24.664755 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:25:24.664737    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4\": container with ID starting with dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4 not found: ID does not exist" containerID="dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4"
Jul 20 10:25:24.664824 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.664763    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4"} err="failed to get container status \"dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4\": rpc error: code = NotFound desc = could not find container \"dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4\": container with ID starting with dcbc8ba90d49c017be21746d9b913523c5171d94cff9c049f92c9b29ffd0d8a4 not found: ID does not exist"
Jul 20 10:25:24.664824 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.664797    2572 scope.go:117] "RemoveContainer" containerID="8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d"
Jul 20 10:25:24.665025 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:25:24.665007    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d\": container with ID starting with 8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d not found: ID does not exist" containerID="8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d"
Jul 20 10:25:24.665064 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.665032    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d"} err="failed to get container status \"8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d\": rpc error: code = NotFound desc = could not find container \"8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d\": container with ID starting with 8dbfb58235f6c408a5eca0e05ce688cfded955f1337aa736c09475788aa2666d not found: ID does not exist"
Jul 20 10:25:24.665064 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.665049    2572 scope.go:117] "RemoveContainer" containerID="d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d"
Jul 20 10:25:24.674509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.674490    2572 scope.go:117] "RemoveContainer" containerID="093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c"
Jul 20 10:25:24.676466 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.676443    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"]
Jul 20 10:25:24.682640 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.682544    2572 scope.go:117] "RemoveContainer" containerID="d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d"
Jul 20 10:25:24.682827 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:25:24.682808    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d\": container with ID starting with d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d not found: ID does not exist" containerID="d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d"
Jul 20 10:25:24.682907 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.682840    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d"} err="failed to get container status \"d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d\": rpc error: code = NotFound desc = could not find container \"d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d\": container with ID starting with d032f24a56e3a07fa8853fb8f5a92b80f8b14edaaea706c4880d02b6e5e42a8d not found: ID does not exist"
Jul 20 10:25:24.682907 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.682856    2572 scope.go:117] "RemoveContainer" containerID="093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c"
Jul 20 10:25:24.683095 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:25:24.683073    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c\": container with ID starting with 093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c not found: ID does not exist" containerID="093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c"
Jul 20 10:25:24.683153 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.683099    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c"} err="failed to get container status \"093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c\": rpc error: code = NotFound desc = could not find container \"093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c\": container with ID starting with 093422569c16c20a0bb2fec5522e71c17fcf66c7841dc5e9c4eaea5d2bf0561c not found: ID does not exist"
Jul 20 10:25:24.683884 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.683864    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetsg2v"]
Jul 20 10:25:24.723569 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.723539    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwbvm\" (UniqueName: \"kubernetes.io/projected/fabba8d3-d52d-4411-bebd-895bc6e208b0-kube-api-access-nwbvm\") pod \"fabba8d3-d52d-4411-bebd-895bc6e208b0\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") "
Jul 20 10:25:24.723712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.723581    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabba8d3-d52d-4411-bebd-895bc6e208b0-tls-certs\") pod \"fabba8d3-d52d-4411-bebd-895bc6e208b0\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") "
Jul 20 10:25:24.723712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.723615    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-model-cache\") pod \"fabba8d3-d52d-4411-bebd-895bc6e208b0\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") "
Jul 20 10:25:24.723712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.723638    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fabba8d3-d52d-4411-bebd-895bc6e208b0-cabundle-cert\") pod \"fabba8d3-d52d-4411-bebd-895bc6e208b0\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") "
Jul 20 10:25:24.723712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.723680    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-home\") pod \"fabba8d3-d52d-4411-bebd-895bc6e208b0\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") "
Jul 20 10:25:24.723712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.723702    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-kserve-provision-location\") pod \"fabba8d3-d52d-4411-bebd-895bc6e208b0\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") "
Jul 20 10:25:24.723973 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.723731    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-dshm\") pod \"fabba8d3-d52d-4411-bebd-895bc6e208b0\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") "
Jul 20 10:25:24.723973 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.723754    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-tmp-dir\") pod \"fabba8d3-d52d-4411-bebd-895bc6e208b0\" (UID: \"fabba8d3-d52d-4411-bebd-895bc6e208b0\") "
Jul 20 10:25:24.724121 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.724063    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabba8d3-d52d-4411-bebd-895bc6e208b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fabba8d3-d52d-4411-bebd-895bc6e208b0" (UID: "fabba8d3-d52d-4411-bebd-895bc6e208b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:25:24.724121 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.724103    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-home" (OuterVolumeSpecName: "home") pod "fabba8d3-d52d-4411-bebd-895bc6e208b0" (UID: "fabba8d3-d52d-4411-bebd-895bc6e208b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:25:24.724291 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.724269    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "fabba8d3-d52d-4411-bebd-895bc6e208b0" (UID: "fabba8d3-d52d-4411-bebd-895bc6e208b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:25:24.726234 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.726202    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabba8d3-d52d-4411-bebd-895bc6e208b0-kube-api-access-nwbvm" (OuterVolumeSpecName: "kube-api-access-nwbvm") pod "fabba8d3-d52d-4411-bebd-895bc6e208b0" (UID: "fabba8d3-d52d-4411-bebd-895bc6e208b0"). InnerVolumeSpecName "kube-api-access-nwbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:25:24.726234 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.726207    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabba8d3-d52d-4411-bebd-895bc6e208b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fabba8d3-d52d-4411-bebd-895bc6e208b0" (UID: "fabba8d3-d52d-4411-bebd-895bc6e208b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:25:24.726393 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.726370    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-dshm" (OuterVolumeSpecName: "dshm") pod "fabba8d3-d52d-4411-bebd-895bc6e208b0" (UID: "fabba8d3-d52d-4411-bebd-895bc6e208b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:25:24.735885 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.735849    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fabba8d3-d52d-4411-bebd-895bc6e208b0" (UID: "fabba8d3-d52d-4411-bebd-895bc6e208b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:25:24.778891 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.778853    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fabba8d3-d52d-4411-bebd-895bc6e208b0" (UID: "fabba8d3-d52d-4411-bebd-895bc6e208b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:25:24.824745 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.824717    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.824745 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.824744    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.824899 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.824754    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.824899 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.824762    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.824899 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.824771    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwbvm\" (UniqueName: \"kubernetes.io/projected/fabba8d3-d52d-4411-bebd-895bc6e208b0-kube-api-access-nwbvm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.824899 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.824778    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabba8d3-d52d-4411-bebd-895bc6e208b0-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.824899 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.824786    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabba8d3-d52d-4411-bebd-895bc6e208b0-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.824899 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.824795    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fabba8d3-d52d-4411-bebd-895bc6e208b0-cabundle-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:25:24.973252 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.973224    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"]
Jul 20 10:25:24.976982 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:24.976959    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6bfb44fd6d-wkt7f"]
Jul 20 10:25:26.522812 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:26.522777    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" path="/var/lib/kubelet/pods/1b57df06-29a1-46f2-b8a5-e162cf21f6b7/volumes"
Jul 20 10:25:26.523213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:26.523200    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" path="/var/lib/kubelet/pods/fabba8d3-d52d-4411-bebd-895bc6e208b0/volumes"
Jul 20 10:25:46.519772 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.519737    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"]
Jul 20 10:25:46.520148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520099    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="storage-initializer"
Jul 20 10:25:46.520148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520113    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="storage-initializer"
Jul 20 10:25:46.520148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520121    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="storage-initializer"
Jul 20 10:25:46.520148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520126    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="storage-initializer"
Jul 20 10:25:46.520148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520138    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main"
Jul 20 10:25:46.520148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520144    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main"
Jul 20 10:25:46.520148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520153    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerName="storage-initializer"
Jul 20 10:25:46.520452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520158    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerName="storage-initializer"
Jul 20 10:25:46.520452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520165    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main"
Jul 20 10:25:46.520452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520170    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main"
Jul 20 10:25:46.520452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520194    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerName="main"
Jul 20 10:25:46.520452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520200    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerName="main"
Jul 20 10:25:46.520452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520242    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b57df06-29a1-46f2-b8a5-e162cf21f6b7" containerName="main"
Jul 20 10:25:46.520452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520251    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb3e14b-e031-4bf9-ab77-1f551e501a89" containerName="main"
Jul 20 10:25:46.520452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.520258    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fabba8d3-d52d-4411-bebd-895bc6e208b0" containerName="main"
Jul 20 10:25:46.523190 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.523173    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.526025 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.526004    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:25:46.526025 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.526017    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 20 10:25:46.526184 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.526044    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-fw5sl\""
Jul 20 10:25:46.526184 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.526021    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 20 10:25:46.526184 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.526057    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 20 10:25:46.527097 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.527082    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 20 10:25:46.537030 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.537007    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"]
Jul 20 10:25:46.597943 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.597907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.598112 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.597971    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.598112 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.598026    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c91e1804-cceb-4a67-a49f-bb1f454364fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.598207 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.598109    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.598207 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.598200    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctht7\" (UniqueName: \"kubernetes.io/projected/c91e1804-cceb-4a67-a49f-bb1f454364fc-kube-api-access-ctht7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.598286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.598231    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91e1804-cceb-4a67-a49f-bb1f454364fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.598286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.598253    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.598390 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.598321    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699244 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699203    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699244 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699248    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c91e1804-cceb-4a67-a49f-bb1f454364fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699296    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctht7\" (UniqueName: \"kubernetes.io/projected/c91e1804-cceb-4a67-a49f-bb1f454364fc-kube-api-access-ctht7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699333    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91e1804-cceb-4a67-a49f-bb1f454364fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699363    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699396    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699424    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699855 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699729    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699855 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699799    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.699855 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699840    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.700010 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.699963    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.700071 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.700047    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c91e1804-cceb-4a67-a49f-bb1f454364fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.701851 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.701823    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.701968 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.701946    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91e1804-cceb-4a67-a49f-bb1f454364fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.708369 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.708337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctht7\" (UniqueName: \"kubernetes.io/projected/c91e1804-cceb-4a67-a49f-bb1f454364fc-kube-api-access-ctht7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.758289 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.758227    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"]
Jul 20 10:25:46.762653 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.762627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:46.766803 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.766787    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-cwqnp\""
Jul 20 10:25:46.774162 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.774105    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"]
Jul 20 10:25:46.833102 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.833068    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:46.901526 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.901491    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:46.901686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.901538    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfp2\" (UniqueName: \"kubernetes.io/projected/0e3e837e-596f-4893-a551-948b41562b7d-kube-api-access-gmfp2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:46.901686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.901651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3e837e-596f-4893-a551-948b41562b7d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:46.901764 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.901716    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:46.969545 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:46.969501    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"]
Jul 20 10:25:46.972629 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:25:46.972600    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91e1804_cceb_4a67_a49f_bb1f454364fc.slice/crio-c1d27d4635d98f7dca3a62d7d1500b000f978feb9ccd01ea556ea7bb10e86f5a WatchSource:0}: Error finding container c1d27d4635d98f7dca3a62d7d1500b000f978feb9ccd01ea556ea7bb10e86f5a: Status 404 returned error can't find the container with id c1d27d4635d98f7dca3a62d7d1500b000f978feb9ccd01ea556ea7bb10e86f5a
Jul 20 10:25:47.002460 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.002433    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.002551 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.002468    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfp2\" (UniqueName: \"kubernetes.io/projected/0e3e837e-596f-4893-a551-948b41562b7d-kube-api-access-gmfp2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.002551 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.002519    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3e837e-596f-4893-a551-948b41562b7d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.002646 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.002593    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.002896 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.002873    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.002896 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.002889    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.004809 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.004788    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3e837e-596f-4893-a551-948b41562b7d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.012317 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.012299    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfp2\" (UniqueName: \"kubernetes.io/projected/0e3e837e-596f-4893-a551-948b41562b7d-kube-api-access-gmfp2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.072712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.072676    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:47.217720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.217694    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"]
Jul 20 10:25:47.219882 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:25:47.219858    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3e837e_596f_4893_a551_948b41562b7d.slice/crio-607a7024edc5db91107b4bcd6346de7ab3cce77fccd51f4f49ed36b09a54ed61 WatchSource:0}: Error finding container 607a7024edc5db91107b4bcd6346de7ab3cce77fccd51f4f49ed36b09a54ed61: Status 404 returned error can't find the container with id 607a7024edc5db91107b4bcd6346de7ab3cce77fccd51f4f49ed36b09a54ed61
Jul 20 10:25:47.742620 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.742584    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" event={"ID":"c91e1804-cceb-4a67-a49f-bb1f454364fc","Type":"ContainerStarted","Data":"ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1"}
Jul 20 10:25:47.743078 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.742652    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" event={"ID":"c91e1804-cceb-4a67-a49f-bb1f454364fc","Type":"ContainerStarted","Data":"c1d27d4635d98f7dca3a62d7d1500b000f978feb9ccd01ea556ea7bb10e86f5a"}
Jul 20 10:25:47.744073 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.744045    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" event={"ID":"0e3e837e-596f-4893-a551-948b41562b7d","Type":"ContainerStarted","Data":"e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2"}
Jul 20 10:25:47.744163 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:47.744093    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" event={"ID":"0e3e837e-596f-4893-a551-948b41562b7d","Type":"ContainerStarted","Data":"607a7024edc5db91107b4bcd6346de7ab3cce77fccd51f4f49ed36b09a54ed61"}
Jul 20 10:25:48.750436 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:48.750402    2572 generic.go:356] "Generic (PLEG): container finished" podID="0e3e837e-596f-4893-a551-948b41562b7d" containerID="e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2" exitCode=0
Jul 20 10:25:48.750837 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:48.750485    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" event={"ID":"0e3e837e-596f-4893-a551-948b41562b7d","Type":"ContainerDied","Data":"e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2"}
Jul 20 10:25:49.756629 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:49.756592    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" event={"ID":"0e3e837e-596f-4893-a551-948b41562b7d","Type":"ContainerStarted","Data":"192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038"}
Jul 20 10:25:49.757177 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:49.756729    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:25:49.792907 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:49.792838    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" podStartSLOduration=3.157950663 podStartE2EDuration="3.792816594s" totalImagesPullingTime="0s" totalInitContainerRuntime="634.865931ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:25:49.791486713 +0000 UTC m=+1729.804588608" watchObservedRunningTime="2026-07-20 10:25:49.792816594 +0000 UTC m=+1729.805918491"
Jul 20 10:25:50.762552 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:50.762522    2572 generic.go:356] "Generic (PLEG): container finished" podID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerID="ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1" exitCode=0
Jul 20 10:25:50.762871 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:50.762589    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" event={"ID":"c91e1804-cceb-4a67-a49f-bb1f454364fc","Type":"ContainerDied","Data":"ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1"}
Jul 20 10:25:51.768458 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:51.768423    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" event={"ID":"c91e1804-cceb-4a67-a49f-bb1f454364fc","Type":"ContainerStarted","Data":"9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47"}
Jul 20 10:25:51.793951 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:51.793891    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podStartSLOduration=2.123325369 podStartE2EDuration="5.793871148s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670545779s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:25:51.791376709 +0000 UTC m=+1731.804478606" watchObservedRunningTime="2026-07-20 10:25:51.793871148 +0000 UTC m=+1731.806973045"
Jul 20 10:25:51.834045 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:51.834007    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:56.834064 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:56.834030    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:25:56.835827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:25:56.835793    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 20 10:26:06.834402 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:26:06.834349    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 20 10:26:16.833914 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:26:16.833862    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 20 10:26:20.764377 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:26:20.764351    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:26:26.834430 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:26:26.834389    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 20 10:26:36.833754 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:26:36.833704    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 20 10:26:46.833921 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:26:46.833876    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 20 10:26:56.834165 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:26:56.834076    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 20 10:27:00.498042 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:00.498014    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:27:00.507101 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:00.507077    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:27:06.843777 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:06.843745    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:27:06.852055 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:06.852027    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:27:08.733402 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:08.733368    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"]
Jul 20 10:27:08.740121 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:08.740095    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"]
Jul 20 10:27:10.764294 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:10.764258    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" podUID="0e3e837e-596f-4893-a551-948b41562b7d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:27:20.765095 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:20.765049    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" podUID="0e3e837e-596f-4893-a551-948b41562b7d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:27:23.754289 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:23.754243    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" podUID="0e3e837e-596f-4893-a551-948b41562b7d" containerName="main" containerID="cri-o://192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038" gracePeriod=45
Jul 20 10:27:23.768935 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:23.768875    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" containerID="cri-o://9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47" gracePeriod=45
Jul 20 10:27:23.844515 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:23.844471    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 20 10:27:24.003189 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.003164    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:27:24.124693 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.124602    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-tokenizer-uds\") pod \"0e3e837e-596f-4893-a551-948b41562b7d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") "
Jul 20 10:27:24.124693 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.124659    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmfp2\" (UniqueName: \"kubernetes.io/projected/0e3e837e-596f-4893-a551-948b41562b7d-kube-api-access-gmfp2\") pod \"0e3e837e-596f-4893-a551-948b41562b7d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") "
Jul 20 10:27:24.124900 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.124707    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3e837e-596f-4893-a551-948b41562b7d-tls-certs\") pod \"0e3e837e-596f-4893-a551-948b41562b7d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") "
Jul 20 10:27:24.124900 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.124770    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-kserve-provision-location\") pod \"0e3e837e-596f-4893-a551-948b41562b7d\" (UID: \"0e3e837e-596f-4893-a551-948b41562b7d\") "
Jul 20 10:27:24.124900 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.124848    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0e3e837e-596f-4893-a551-948b41562b7d" (UID: "0e3e837e-596f-4893-a551-948b41562b7d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:27:24.125048 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.125029    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.125545 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.125519    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e3e837e-596f-4893-a551-948b41562b7d" (UID: "0e3e837e-596f-4893-a551-948b41562b7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:27:24.127013 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.126986    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3e837e-596f-4893-a551-948b41562b7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e3e837e-596f-4893-a551-948b41562b7d" (UID: "0e3e837e-596f-4893-a551-948b41562b7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:27:24.127118 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.127095    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3e837e-596f-4893-a551-948b41562b7d-kube-api-access-gmfp2" (OuterVolumeSpecName: "kube-api-access-gmfp2") pod "0e3e837e-596f-4893-a551-948b41562b7d" (UID: "0e3e837e-596f-4893-a551-948b41562b7d"). InnerVolumeSpecName "kube-api-access-gmfp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:27:24.161508 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.161478    2572 generic.go:356] "Generic (PLEG): container finished" podID="0e3e837e-596f-4893-a551-948b41562b7d" containerID="192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038" exitCode=0
Jul 20 10:27:24.161623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.161543    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"
Jul 20 10:27:24.161623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.161591    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" event={"ID":"0e3e837e-596f-4893-a551-948b41562b7d","Type":"ContainerDied","Data":"192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038"}
Jul 20 10:27:24.161745 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.161653    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d" event={"ID":"0e3e837e-596f-4893-a551-948b41562b7d","Type":"ContainerDied","Data":"607a7024edc5db91107b4bcd6346de7ab3cce77fccd51f4f49ed36b09a54ed61"}
Jul 20 10:27:24.161745 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.161685    2572 scope.go:117] "RemoveContainer" containerID="192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038"
Jul 20 10:27:24.208866 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.208842    2572 scope.go:117] "RemoveContainer" containerID="e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2"
Jul 20 10:27:24.217826 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.217804    2572 scope.go:117] "RemoveContainer" containerID="192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038"
Jul 20 10:27:24.218120 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:27:24.218100    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038\": container with ID starting with 192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038 not found: ID does not exist" containerID="192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038"
Jul 20 10:27:24.218181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.218128    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038"} err="failed to get container status \"192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038\": rpc error: code = NotFound desc = could not find container \"192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038\": container with ID starting with 192a444bff5f8abf617cee7a2cb9799ca85fb3a87b5085117300c4a1cf7e3038 not found: ID does not exist"
Jul 20 10:27:24.218181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.218146    2572 scope.go:117] "RemoveContainer" containerID="e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2"
Jul 20 10:27:24.218414 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:27:24.218398    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2\": container with ID starting with e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2 not found: ID does not exist" containerID="e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2"
Jul 20 10:27:24.218473 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.218418    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2"} err="failed to get container status \"e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2\": rpc error: code = NotFound desc = could not find container \"e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2\": container with ID starting with e22eb4b34b63939968b269611ba8355bcb20e930de299cfe66b59e7a616b97f2 not found: ID does not exist"
Jul 20 10:27:24.222856 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.222835    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"]
Jul 20 10:27:24.225998 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.225964    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3e837e-596f-4893-a551-948b41562b7d-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.225998 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.225993    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmfp2\" (UniqueName: \"kubernetes.io/projected/0e3e837e-596f-4893-a551-948b41562b7d-kube-api-access-gmfp2\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.226143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.226008    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3e837e-596f-4893-a551-948b41562b7d-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.227596 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.227572    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches896d"]
Jul 20 10:27:24.327816 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.327793    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:27:24.427574 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.427479    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91e1804-cceb-4a67-a49f-bb1f454364fc-tls-certs\") pod \"c91e1804-cceb-4a67-a49f-bb1f454364fc\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") "
Jul 20 10:27:24.427574 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.427535    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctht7\" (UniqueName: \"kubernetes.io/projected/c91e1804-cceb-4a67-a49f-bb1f454364fc-kube-api-access-ctht7\") pod \"c91e1804-cceb-4a67-a49f-bb1f454364fc\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") "
Jul 20 10:27:24.427753 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.427608    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c91e1804-cceb-4a67-a49f-bb1f454364fc-cabundle-cert\") pod \"c91e1804-cceb-4a67-a49f-bb1f454364fc\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") "
Jul 20 10:27:24.427753 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.427647    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-tmp-dir\") pod \"c91e1804-cceb-4a67-a49f-bb1f454364fc\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") "
Jul 20 10:27:24.427753 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.427681    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-kserve-provision-location\") pod \"c91e1804-cceb-4a67-a49f-bb1f454364fc\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") "
Jul 20 10:27:24.427753 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.427714    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-home\") pod \"c91e1804-cceb-4a67-a49f-bb1f454364fc\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") "
Jul 20 10:27:24.427950 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.427771    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-model-cache\") pod \"c91e1804-cceb-4a67-a49f-bb1f454364fc\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") "
Jul 20 10:27:24.427950 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.427827    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-dshm\") pod \"c91e1804-cceb-4a67-a49f-bb1f454364fc\" (UID: \"c91e1804-cceb-4a67-a49f-bb1f454364fc\") "
Jul 20 10:27:24.428100 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.428075    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c91e1804-cceb-4a67-a49f-bb1f454364fc" (UID: "c91e1804-cceb-4a67-a49f-bb1f454364fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:27:24.428100 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.428084    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91e1804-cceb-4a67-a49f-bb1f454364fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c91e1804-cceb-4a67-a49f-bb1f454364fc" (UID: "c91e1804-cceb-4a67-a49f-bb1f454364fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:27:24.428213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.428172    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c91e1804-cceb-4a67-a49f-bb1f454364fc" (UID: "c91e1804-cceb-4a67-a49f-bb1f454364fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:27:24.428268 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.428212    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-home" (OuterVolumeSpecName: "home") pod "c91e1804-cceb-4a67-a49f-bb1f454364fc" (UID: "c91e1804-cceb-4a67-a49f-bb1f454364fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:27:24.428371 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.428353    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.428433 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.428377    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c91e1804-cceb-4a67-a49f-bb1f454364fc-cabundle-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.428433 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.428392    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.428433 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.428404    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.429768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.429744    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91e1804-cceb-4a67-a49f-bb1f454364fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c91e1804-cceb-4a67-a49f-bb1f454364fc" (UID: "c91e1804-cceb-4a67-a49f-bb1f454364fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:27:24.429853 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.429837    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91e1804-cceb-4a67-a49f-bb1f454364fc-kube-api-access-ctht7" (OuterVolumeSpecName: "kube-api-access-ctht7") pod "c91e1804-cceb-4a67-a49f-bb1f454364fc" (UID: "c91e1804-cceb-4a67-a49f-bb1f454364fc"). InnerVolumeSpecName "kube-api-access-ctht7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:27:24.430231 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.430215    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-dshm" (OuterVolumeSpecName: "dshm") pod "c91e1804-cceb-4a67-a49f-bb1f454364fc" (UID: "c91e1804-cceb-4a67-a49f-bb1f454364fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:27:24.492933 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.492897    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c91e1804-cceb-4a67-a49f-bb1f454364fc" (UID: "c91e1804-cceb-4a67-a49f-bb1f454364fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:27:24.519677 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.519648    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e3e837e-596f-4893-a551-948b41562b7d" path="/var/lib/kubelet/pods/0e3e837e-596f-4893-a551-948b41562b7d/volumes"
Jul 20 10:27:24.528737 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.528720    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.528737 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.528738    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91e1804-cceb-4a67-a49f-bb1f454364fc-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.528854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.528747    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctht7\" (UniqueName: \"kubernetes.io/projected/c91e1804-cceb-4a67-a49f-bb1f454364fc-kube-api-access-ctht7\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:24.528854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:24.528756    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c91e1804-cceb-4a67-a49f-bb1f454364fc-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:25.167773 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.167689    2572 generic.go:356] "Generic (PLEG): container finished" podID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerID="9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47" exitCode=0
Jul 20 10:27:25.167773 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.167743    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"
Jul 20 10:27:25.168231 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.167748    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" event={"ID":"c91e1804-cceb-4a67-a49f-bb1f454364fc","Type":"ContainerDied","Data":"9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47"}
Jul 20 10:27:25.168231 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.167853    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk" event={"ID":"c91e1804-cceb-4a67-a49f-bb1f454364fc","Type":"ContainerDied","Data":"c1d27d4635d98f7dca3a62d7d1500b000f978feb9ccd01ea556ea7bb10e86f5a"}
Jul 20 10:27:25.168231 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.167875    2572 scope.go:117] "RemoveContainer" containerID="9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47"
Jul 20 10:27:25.175691 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.175671    2572 scope.go:117] "RemoveContainer" containerID="ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1"
Jul 20 10:27:25.183497 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.183478    2572 scope.go:117] "RemoveContainer" containerID="9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47"
Jul 20 10:27:25.183757 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:27:25.183735    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47\": container with ID starting with 9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47 not found: ID does not exist" containerID="9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47"
Jul 20 10:27:25.183809 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.183766    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47"} err="failed to get container status \"9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47\": rpc error: code = NotFound desc = could not find container \"9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47\": container with ID starting with 9431c4f6637ed19cf7068a419fc6bad2cf47b6456e7ba0ea40a7000abaa33a47 not found: ID does not exist"
Jul 20 10:27:25.183809 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.183783    2572 scope.go:117] "RemoveContainer" containerID="ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1"
Jul 20 10:27:25.184031 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:27:25.184012    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1\": container with ID starting with ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1 not found: ID does not exist" containerID="ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1"
Jul 20 10:27:25.184073 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.184037    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1"} err="failed to get container status \"ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1\": rpc error: code = NotFound desc = could not find container \"ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1\": container with ID starting with ef857d7adb908d577705e69242abc06e3034ba69fc9e70c94914420a9b2fe5a1 not found: ID does not exist"
Jul 20 10:27:25.189945 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.189925    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"]
Jul 20 10:27:25.194044 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:25.194023    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-ddd7cdf75-dhqvk"]
Jul 20 10:27:26.519655 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:26.519624    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" path="/var/lib/kubelet/pods/c91e1804-cceb-4a67-a49f-bb1f454364fc/volumes"
Jul 20 10:27:31.961077 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961040    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"]
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961347    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3e837e-596f-4893-a551-948b41562b7d" containerName="main"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961359    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3e837e-596f-4893-a551-948b41562b7d" containerName="main"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961371    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="storage-initializer"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961376    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="storage-initializer"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961383    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961388    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961400    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3e837e-596f-4893-a551-948b41562b7d" containerName="storage-initializer"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961405    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3e837e-596f-4893-a551-948b41562b7d" containerName="storage-initializer"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961471    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3e837e-596f-4893-a551-948b41562b7d" containerName="main"
Jul 20 10:27:31.961520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.961478    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c91e1804-cceb-4a67-a49f-bb1f454364fc" containerName="main"
Jul 20 10:27:31.963973 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.963957    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:31.966636 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.966613    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 20 10:27:31.966752 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.966620    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 20 10:27:31.967808 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.967789    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-p6sgc\""
Jul 20 10:27:31.969776 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:31.969752    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"]
Jul 20 10:27:32.095428 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:32.095395    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65vg\" (UniqueName: \"kubernetes.io/projected/e38c701a-d4fa-406a-8b04-96e5219efee8-kube-api-access-r65vg\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:32.196155 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:32.196116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65vg\" (UniqueName: \"kubernetes.io/projected/e38c701a-d4fa-406a-8b04-96e5219efee8-kube-api-access-r65vg\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:32.204413 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:32.204380    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65vg\" (UniqueName: \"kubernetes.io/projected/e38c701a-d4fa-406a-8b04-96e5219efee8-kube-api-access-r65vg\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.145594 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.145528    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.154046 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.154020    2572 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.246897 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.246861    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.247045 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.246972    2572 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.250748 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.250727    2572 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") DevicePath \"csi-1dcffb661c0108a699b08094446b512de1eff7c05bd379a0d7c08a14ab827757\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.467832 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.467788    2572 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/bf18d13d69e58b44b1d5cc4e365da380702fed80e36a7ea2cd1410602850eb40/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.480949 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.480919    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"e2e-pvc-model-download-hnvrl\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.573159 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.573127    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:38.701437 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:38.701414    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"]
Jul 20 10:27:38.703867 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:27:38.703839    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode38c701a_d4fa_406a_8b04_96e5219efee8.slice/crio-cd04e6b50943a9ca2cbf0ae230a5c631fae713cbdb3634112528e7630bfcda07 WatchSource:0}: Error finding container cd04e6b50943a9ca2cbf0ae230a5c631fae713cbdb3634112528e7630bfcda07: Status 404 returned error can't find the container with id cd04e6b50943a9ca2cbf0ae230a5c631fae713cbdb3634112528e7630bfcda07
Jul 20 10:27:39.225576 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:39.225519    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl" event={"ID":"e38c701a-d4fa-406a-8b04-96e5219efee8","Type":"ContainerStarted","Data":"8a0b6d01ab70bfde81a41671fb3dbd00c2e35b7abecea3f177441f54c5ab9bf0"}
Jul 20 10:27:39.225576 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:39.225581    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl" event={"ID":"e38c701a-d4fa-406a-8b04-96e5219efee8","Type":"ContainerStarted","Data":"cd04e6b50943a9ca2cbf0ae230a5c631fae713cbdb3634112528e7630bfcda07"}
Jul 20 10:27:39.242186 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:39.242139    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl" podStartSLOduration=11.242125794 podStartE2EDuration="11.242125794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:27:39.241741016 +0000 UTC m=+1839.254842913" watchObservedRunningTime="2026-07-20 10:27:39.242125794 +0000 UTC m=+1839.255227735"
Jul 20 10:27:45.251706 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:45.251674    2572 generic.go:356] "Generic (PLEG): container finished" podID="e38c701a-d4fa-406a-8b04-96e5219efee8" containerID="8a0b6d01ab70bfde81a41671fb3dbd00c2e35b7abecea3f177441f54c5ab9bf0" exitCode=0
Jul 20 10:27:45.252146 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:45.251747    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl" event={"ID":"e38c701a-d4fa-406a-8b04-96e5219efee8","Type":"ContainerDied","Data":"8a0b6d01ab70bfde81a41671fb3dbd00c2e35b7abecea3f177441f54c5ab9bf0"}
Jul 20 10:27:46.390858 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.390836    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:46.517644 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.517540    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r65vg\" (UniqueName: \"kubernetes.io/projected/e38c701a-d4fa-406a-8b04-96e5219efee8-kube-api-access-r65vg\") pod \"e38c701a-d4fa-406a-8b04-96e5219efee8\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") "
Jul 20 10:27:46.517814 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.517668    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"e38c701a-d4fa-406a-8b04-96e5219efee8\" (UID: \"e38c701a-d4fa-406a-8b04-96e5219efee8\") "
Jul 20 10:27:46.524984 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.524945    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38c701a-d4fa-406a-8b04-96e5219efee8-kube-api-access-r65vg" (OuterVolumeSpecName: "kube-api-access-r65vg") pod "e38c701a-d4fa-406a-8b04-96e5219efee8" (UID: "e38c701a-d4fa-406a-8b04-96e5219efee8"). InnerVolumeSpecName "kube-api-access-r65vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:27:46.525440 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.525414    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac" (OuterVolumeSpecName: "model-storage") pod "e38c701a-d4fa-406a-8b04-96e5219efee8" (UID: "e38c701a-d4fa-406a-8b04-96e5219efee8"). InnerVolumeSpecName "pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 20 10:27:46.619008 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.618958    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r65vg\" (UniqueName: \"kubernetes.io/projected/e38c701a-d4fa-406a-8b04-96e5219efee8-kube-api-access-r65vg\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:27:46.619008 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.619018    2572 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") on node \"ip-10-0-138-196.ec2.internal\" "
Jul 20 10:27:46.739829 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.739803    2572 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac") on node "ip-10-0-138-196.ec2.internal"
Jul 20 10:27:46.820700 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:46.820609    2572 reconciler_common.go:299] "Volume detached for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"csi-1dcffb661c0108a699b08094446b512de1eff7c05bd379a0d7c08a14ab827757\""
Jul 20 10:27:47.261610 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:47.261581    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"
Jul 20 10:27:47.261779 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:47.261586    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl" event={"ID":"e38c701a-d4fa-406a-8b04-96e5219efee8","Type":"ContainerDied","Data":"cd04e6b50943a9ca2cbf0ae230a5c631fae713cbdb3634112528e7630bfcda07"}
Jul 20 10:27:47.261779 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:27:47.261685    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd04e6b50943a9ca2cbf0ae230a5c631fae713cbdb3634112528e7630bfcda07"
Jul 20 10:28:01.296717 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.296686    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"]
Jul 20 10:28:01.297096 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.296973    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e38c701a-d4fa-406a-8b04-96e5219efee8" containerName="storage-initializer"
Jul 20 10:28:01.297096 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.296984    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e38c701a-d4fa-406a-8b04-96e5219efee8" containerName="storage-initializer"
Jul 20 10:28:01.297096 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.297038    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e38c701a-d4fa-406a-8b04-96e5219efee8" containerName="storage-initializer"
Jul 20 10:28:01.301286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.301267    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.304021 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.303998    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 20 10:28:01.304150 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.304106    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 20 10:28:01.304150 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.304106    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-p6sgc\""
Jul 20 10:28:01.304262 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.304238    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 20 10:28:01.312046 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.312022    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"]
Jul 20 10:28:01.327849 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.327822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.327995 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.327903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.327995 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.327957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.327995 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.327985    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.328144 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.328009    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429322 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429283    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429322 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429377    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnbc\" (UniqueName: \"kubernetes.io/projected/8cd136fa-971c-494e-92eb-b6ba5fc692c5-kube-api-access-7mnbc\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429406    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429509    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429841 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429820    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429910 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429868    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.429910 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.429887    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.431567 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.431537    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.431862 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.431844    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.530128 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.530091    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnbc\" (UniqueName: \"kubernetes.io/projected/8cd136fa-971c-494e-92eb-b6ba5fc692c5-kube-api-access-7mnbc\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:01.549532 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:01.549459    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnbc\" (UniqueName: \"kubernetes.io/projected/8cd136fa-971c-494e-92eb-b6ba5fc692c5-kube-api-access-7mnbc\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:08.589458 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:08.589414    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:08.606120 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:08.606093    2572 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:08.690018 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:08.689982    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:08.690172 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:08.690092    2572 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:08.694515 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:08.694492    2572 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") DevicePath \"csi-1dcffb661c0108a699b08094446b512de1eff7c05bd379a0d7c08a14ab827757\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:08.823460 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:08.823421    2572 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/bf18d13d69e58b44b1d5cc4e365da380702fed80e36a7ea2cd1410602850eb40/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:08.841805 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:08.841746    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:09.112134 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:09.112042    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:09.243498 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:09.243467    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"]
Jul 20 10:28:09.246978 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:28:09.246950    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd136fa_971c_494e_92eb_b6ba5fc692c5.slice/crio-fff448167a17b2606c9bcce771f7e51f3f34d978758391cae54b3fff0fd09641 WatchSource:0}: Error finding container fff448167a17b2606c9bcce771f7e51f3f34d978758391cae54b3fff0fd09641: Status 404 returned error can't find the container with id fff448167a17b2606c9bcce771f7e51f3f34d978758391cae54b3fff0fd09641
Jul 20 10:28:09.351452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:09.351417    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" event={"ID":"8cd136fa-971c-494e-92eb-b6ba5fc692c5","Type":"ContainerStarted","Data":"cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b"}
Jul 20 10:28:09.351641 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:09.351474    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" event={"ID":"8cd136fa-971c-494e-92eb-b6ba5fc692c5","Type":"ContainerStarted","Data":"fff448167a17b2606c9bcce771f7e51f3f34d978758391cae54b3fff0fd09641"}
Jul 20 10:28:09.371950 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:09.371858    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podStartSLOduration=8.371843111 podStartE2EDuration="8.371843111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:28:09.370088175 +0000 UTC m=+1869.383190098" watchObservedRunningTime="2026-07-20 10:28:09.371843111 +0000 UTC m=+1869.384945047"
Jul 20 10:28:10.113060 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:10.113024    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:19.112323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:19.112268    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:28:19.113737 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:19.113706    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:28:29.113229 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:29.113181    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:28:39.113697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:39.113637    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:28:49.113035 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:49.112983    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:28:59.113516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:28:59.113475    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:29:09.112842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:29:09.112791    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:29:19.113219 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:29:19.113153    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:29:29.113240 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:29:29.113198    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:29:39.113015 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:29:39.112962    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:29:49.112778 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:29:49.112734    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:29:59.113320 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:29:59.113215    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:30:09.122362 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:09.122327    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:30:09.130167 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:09.130142    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:30:13.998516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:13.998478    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"]
Jul 20 10:30:29.021501 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.021459    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" containerID="cri-o://cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b" gracePeriod=45
Jul 20 10:30:29.123589 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.123525    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 20 10:30:29.769442 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.769421    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:30:29.884725 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.884632    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-home\") pod \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") "
Jul 20 10:30:29.884894 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.884762    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") pod \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") "
Jul 20 10:30:29.884894 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.884791    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-dshm\") pod \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") "
Jul 20 10:30:29.884894 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.884852    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tmp-dir\") pod \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") "
Jul 20 10:30:29.884894 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.884885    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-model-cache\") pod \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") "
Jul 20 10:30:29.885099 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.884921    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tls-certs\") pod \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") "
Jul 20 10:30:29.885099 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.884973    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mnbc\" (UniqueName: \"kubernetes.io/projected/8cd136fa-971c-494e-92eb-b6ba5fc692c5-kube-api-access-7mnbc\") pod \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\" (UID: \"8cd136fa-971c-494e-92eb-b6ba5fc692c5\") "
Jul 20 10:30:29.885224 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.885162    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "8cd136fa-971c-494e-92eb-b6ba5fc692c5" (UID: "8cd136fa-971c-494e-92eb-b6ba5fc692c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:30:29.885520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.885489    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-home" (OuterVolumeSpecName: "home") pod "8cd136fa-971c-494e-92eb-b6ba5fc692c5" (UID: "8cd136fa-971c-494e-92eb-b6ba5fc692c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:30:29.887325 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.887301    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd136fa-971c-494e-92eb-b6ba5fc692c5-kube-api-access-7mnbc" (OuterVolumeSpecName: "kube-api-access-7mnbc") pod "8cd136fa-971c-494e-92eb-b6ba5fc692c5" (UID: "8cd136fa-971c-494e-92eb-b6ba5fc692c5"). InnerVolumeSpecName "kube-api-access-7mnbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:30:29.887612 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.887587    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cd136fa-971c-494e-92eb-b6ba5fc692c5" (UID: "8cd136fa-971c-494e-92eb-b6ba5fc692c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:30:29.887612 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.887609    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-dshm" (OuterVolumeSpecName: "dshm") pod "8cd136fa-971c-494e-92eb-b6ba5fc692c5" (UID: "8cd136fa-971c-494e-92eb-b6ba5fc692c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:30:29.887743 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.887621    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac" (OuterVolumeSpecName: "kserve-pvc-source") pod "8cd136fa-971c-494e-92eb-b6ba5fc692c5" (UID: "8cd136fa-971c-494e-92eb-b6ba5fc692c5"). InnerVolumeSpecName "pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 20 10:30:29.897071 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.897050    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8cd136fa-971c-494e-92eb-b6ba5fc692c5" (UID: "8cd136fa-971c-494e-92eb-b6ba5fc692c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:30:29.937028 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.936999    2572 generic.go:356] "Generic (PLEG): container finished" podID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerID="cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b" exitCode=0
Jul 20 10:30:29.937123 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.937051    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"
Jul 20 10:30:29.937123 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.937068    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" event={"ID":"8cd136fa-971c-494e-92eb-b6ba5fc692c5","Type":"ContainerDied","Data":"cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b"}
Jul 20 10:30:29.937123 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.937120    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n" event={"ID":"8cd136fa-971c-494e-92eb-b6ba5fc692c5","Type":"ContainerDied","Data":"fff448167a17b2606c9bcce771f7e51f3f34d978758391cae54b3fff0fd09641"}
Jul 20 10:30:29.937260 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.937158    2572 scope.go:117] "RemoveContainer" containerID="cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b"
Jul 20 10:30:29.945322 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.945305    2572 scope.go:117] "RemoveContainer" containerID="cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b"
Jul 20 10:30:29.945590 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:30:29.945549    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b\": container with ID starting with cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b not found: ID does not exist" containerID="cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b"
Jul 20 10:30:29.945678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.945601    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b"} err="failed to get container status \"cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b\": rpc error: code = NotFound desc = could not find container \"cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b\": container with ID starting with cce9c1bc1d0e12964862f4be94f10cb35fa13e29d99a200090b6a9dd3f1c8e9b not found: ID does not exist"
Jul 20 10:30:29.958211 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.958189    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"]
Jul 20 10:30:29.961386 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.961367    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6764dbdddbsqz6n"]
Jul 20 10:30:29.985509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.985483    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:30:29.985509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.985510    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:30:29.985695 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.985519    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd136fa-971c-494e-92eb-b6ba5fc692c5-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:30:29.985695 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.985527    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mnbc\" (UniqueName: \"kubernetes.io/projected/8cd136fa-971c-494e-92eb-b6ba5fc692c5-kube-api-access-7mnbc\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:30:29.985695 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.985536    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:30:29.985695 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.985581    2572 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") on node \"ip-10-0-138-196.ec2.internal\" "
Jul 20 10:30:29.985695 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:29.985593    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cd136fa-971c-494e-92eb-b6ba5fc692c5-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:30:30.011022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:30.011000    2572 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac") on node "ip-10-0-138-196.ec2.internal"
Jul 20 10:30:30.086978 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:30.086933    2572 reconciler_common.go:299] "Volume detached for volume \"pvc-3f2a219e-b3d9-41aa-9e66-694dc12f1669\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00ef406e4ac589bac\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"csi-1dcffb661c0108a699b08094446b512de1eff7c05bd379a0d7c08a14ab827757\""
Jul 20 10:30:30.520445 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:30.520408    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" path="/var/lib/kubelet/pods/8cd136fa-971c-494e-92eb-b6ba5fc692c5/volumes"
Jul 20 10:30:37.272526 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.272488    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"]
Jul 20 10:30:37.273021 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.272827    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main"
Jul 20 10:30:37.273021 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.272839    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main"
Jul 20 10:30:37.273021 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.272895    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd136fa-971c-494e-92eb-b6ba5fc692c5" containerName="main"
Jul 20 10:30:37.275739 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.275722    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:37.278710 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.278680    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 20 10:30:37.279826 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.279797    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 20 10:30:37.279826 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.279807    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-8455f\""
Jul 20 10:30:37.283137 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.283116    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"]
Jul 20 10:30:37.346484 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.346447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ns6\" (UniqueName: \"kubernetes.io/projected/6f0f3e23-1b34-4eb9-846d-4082355c5886-kube-api-access-b8ns6\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:37.447927 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.447895    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ns6\" (UniqueName: \"kubernetes.io/projected/6f0f3e23-1b34-4eb9-846d-4082355c5886-kube-api-access-b8ns6\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:37.458797 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:37.458766    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ns6\" (UniqueName: \"kubernetes.io/projected/6f0f3e23-1b34-4eb9-846d-4082355c5886-kube-api-access-b8ns6\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:38.950535 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:38.950499    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"]
Jul 20 10:30:38.954300 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:38.954275    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hnvrl"]
Jul 20 10:30:40.520262 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:40.520226    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e38c701a-d4fa-406a-8b04-96e5219efee8" path="/var/lib/kubelet/pods/e38c701a-d4fa-406a-8b04-96e5219efee8/volumes"
Jul 20 10:30:40.674768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:40.674725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:40.681311 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:40.681281    2572 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:40.775783 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:40.775685    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:40.775937 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:40.775816    2572 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:40.779377 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:40.779355    2572 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") DevicePath \"csi-96f8f215107b6e036516bdbf320e5d4c7f0cf98c203f1213a91942cce6bf9133\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:40.995017 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:40.994988    2572 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d05f987163083a2c1e92bc2237b87aeb4cab9180ea8a17abb3979848d36baac2/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:41.008127 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:41.008101    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"e2e-pvc-model-download-gqlt6\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:41.185707 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:41.185623    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:41.315880 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:41.315841    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"]
Jul 20 10:30:41.318764 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:30:41.318736    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f0f3e23_1b34_4eb9_846d_4082355c5886.slice/crio-8343709f1e20be4063827d14d27e99d625ccaafaf137a7ef6eedfe0daac92797 WatchSource:0}: Error finding container 8343709f1e20be4063827d14d27e99d625ccaafaf137a7ef6eedfe0daac92797: Status 404 returned error can't find the container with id 8343709f1e20be4063827d14d27e99d625ccaafaf137a7ef6eedfe0daac92797
Jul 20 10:30:41.985086 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:41.985052    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6" event={"ID":"6f0f3e23-1b34-4eb9-846d-4082355c5886","Type":"ContainerStarted","Data":"07b0d938103ef1e3d79dc9afed9eba3afca5d89c49b41e145cd0c6d24d9eb1b9"}
Jul 20 10:30:41.985528 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:41.985097    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6" event={"ID":"6f0f3e23-1b34-4eb9-846d-4082355c5886","Type":"ContainerStarted","Data":"8343709f1e20be4063827d14d27e99d625ccaafaf137a7ef6eedfe0daac92797"}
Jul 20 10:30:42.002507 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:42.002454    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6" podStartSLOduration=8.002437879 podStartE2EDuration="8.002437879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:30:42.002219441 +0000 UTC m=+2022.015321337" watchObservedRunningTime="2026-07-20 10:30:42.002437879 +0000 UTC m=+2022.015539774"
Jul 20 10:30:48.010768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:48.010730    2572 generic.go:356] "Generic (PLEG): container finished" podID="6f0f3e23-1b34-4eb9-846d-4082355c5886" containerID="07b0d938103ef1e3d79dc9afed9eba3afca5d89c49b41e145cd0c6d24d9eb1b9" exitCode=0
Jul 20 10:30:48.011250 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:48.010774    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6" event={"ID":"6f0f3e23-1b34-4eb9-846d-4082355c5886","Type":"ContainerDied","Data":"07b0d938103ef1e3d79dc9afed9eba3afca5d89c49b41e145cd0c6d24d9eb1b9"}
Jul 20 10:30:49.165753 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.165728    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:49.251479 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.251438    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8ns6\" (UniqueName: \"kubernetes.io/projected/6f0f3e23-1b34-4eb9-846d-4082355c5886-kube-api-access-b8ns6\") pod \"6f0f3e23-1b34-4eb9-846d-4082355c5886\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") "
Jul 20 10:30:49.251670 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.251624    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"6f0f3e23-1b34-4eb9-846d-4082355c5886\" (UID: \"6f0f3e23-1b34-4eb9-846d-4082355c5886\") "
Jul 20 10:30:49.253711 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.253682    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0f3e23-1b34-4eb9-846d-4082355c5886-kube-api-access-b8ns6" (OuterVolumeSpecName: "kube-api-access-b8ns6") pod "6f0f3e23-1b34-4eb9-846d-4082355c5886" (UID: "6f0f3e23-1b34-4eb9-846d-4082355c5886"). InnerVolumeSpecName "kube-api-access-b8ns6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:30:49.254319 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.254300    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed" (OuterVolumeSpecName: "model-storage") pod "6f0f3e23-1b34-4eb9-846d-4082355c5886" (UID: "6f0f3e23-1b34-4eb9-846d-4082355c5886"). InnerVolumeSpecName "pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 20 10:30:49.352852 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.352746    2572 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") on node \"ip-10-0-138-196.ec2.internal\" "
Jul 20 10:30:49.352852 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.352782    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8ns6\" (UniqueName: \"kubernetes.io/projected/6f0f3e23-1b34-4eb9-846d-4082355c5886-kube-api-access-b8ns6\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:30:49.470820 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.470797    2572 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed") on node "ip-10-0-138-196.ec2.internal"
Jul 20 10:30:49.554389 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:49.554342    2572 reconciler_common.go:299] "Volume detached for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"csi-96f8f215107b6e036516bdbf320e5d4c7f0cf98c203f1213a91942cce6bf9133\""
Jul 20 10:30:50.020885 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:50.020848    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6" event={"ID":"6f0f3e23-1b34-4eb9-846d-4082355c5886","Type":"ContainerDied","Data":"8343709f1e20be4063827d14d27e99d625ccaafaf137a7ef6eedfe0daac92797"}
Jul 20 10:30:50.020885 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:50.020880    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"
Jul 20 10:30:50.020885 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:30:50.020891    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8343709f1e20be4063827d14d27e99d625ccaafaf137a7ef6eedfe0daac92797"
Jul 20 10:31:03.505447 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.505412    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"]
Jul 20 10:31:03.505871 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.505732    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0f3e23-1b34-4eb9-846d-4082355c5886" containerName="storage-initializer"
Jul 20 10:31:03.505871 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.505744    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0f3e23-1b34-4eb9-846d-4082355c5886" containerName="storage-initializer"
Jul 20 10:31:03.505871 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.505792    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f0f3e23-1b34-4eb9-846d-4082355c5886" containerName="storage-initializer"
Jul 20 10:31:03.510234 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.510210    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.513131 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.513100    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 20 10:31:03.513372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.513354    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-8455f\""
Jul 20 10:31:03.513424 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.513408    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 20 10:31:03.513699 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.513682    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 20 10:31:03.514830 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.514808    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-7c8jt\""
Jul 20 10:31:03.516701 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.516680    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"]
Jul 20 10:31:03.519871 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.519854    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.523876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.523849    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"]
Jul 20 10:31:03.542133 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.542096    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"]
Jul 20 10:31:03.671423 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.671423 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671430    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352b76fd-c536-469a-a96b-0b7df702b444-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.671720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671463    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31bb17e-3044-4925-8c5e-b2ec82810be4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.671720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.671720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.671720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671641    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.671720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrgf\" (UniqueName: \"kubernetes.io/projected/a31bb17e-3044-4925-8c5e-b2ec82810be4-kube-api-access-dsrgf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.671720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671690    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.671720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qcs\" (UniqueName: \"kubernetes.io/projected/352b76fd-c536-469a-a96b-0b7df702b444-kube-api-access-z4qcs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.671981 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.671981 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671756    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.671981 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.671803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773254 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773146    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrgf\" (UniqueName: \"kubernetes.io/projected/a31bb17e-3044-4925-8c5e-b2ec82810be4-kube-api-access-dsrgf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773254 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773197    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.773254 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773234    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qcs\" (UniqueName: \"kubernetes.io/projected/352b76fd-c536-469a-a96b-0b7df702b444-kube-api-access-z4qcs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.773540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773283    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.773540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773394    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.773540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773417    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352b76fd-c536-469a-a96b-0b7df702b444-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.773540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773447    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31bb17e-3044-4925-8c5e-b2ec82810be4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773486    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773540 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773521    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773992 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773550    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.773992 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773764    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773992 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773805    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.773992 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773806    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.773992 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.773888    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.774245 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.774020    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.774245 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.774078    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.775940 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.775912    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.776094 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.776073    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.776252 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.776227    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352b76fd-c536-469a-a96b-0b7df702b444-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:03.776356 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.776340    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31bb17e-3044-4925-8c5e-b2ec82810be4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.791854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.791822    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrgf\" (UniqueName: \"kubernetes.io/projected/a31bb17e-3044-4925-8c5e-b2ec82810be4-kube-api-access-dsrgf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:03.791854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:03.791851    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qcs\" (UniqueName: \"kubernetes.io/projected/352b76fd-c536-469a-a96b-0b7df702b444-kube-api-access-z4qcs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.336133 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.336093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.343413 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.342988    2572 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.436686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.436644    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.436869 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.436788    2572 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.441089 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.441065    2572 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") DevicePath \"csi-96f8f215107b6e036516bdbf320e5d4c7f0cf98c203f1213a91942cce6bf9133\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.534495 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.534446    2572 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d05f987163083a2c1e92bc2237b87aeb4cab9180ea8a17abb3979848d36baac2/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.550234 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.550203    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.619796 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.619695    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:11.639378 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.639328    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:11.639516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.639467    2572 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") DevicePath \"csi-96f8f215107b6e036516bdbf320e5d4c7f0cf98c203f1213a91942cce6bf9133\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:11.643341 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.643313    2572 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") DevicePath \"csi-96f8f215107b6e036516bdbf320e5d4c7f0cf98c203f1213a91942cce6bf9133\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:11.658585 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.658541    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:11.762815 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.762787    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"]
Jul 20 10:31:11.765312 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:31:11.765285    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352b76fd_c536_469a_a96b_0b7df702b444.slice/crio-4b3f6f5e2076b021f0bda9a71caa974e7ea2db1ee25fa9de86cb2d7de26f360e WatchSource:0}: Error finding container 4b3f6f5e2076b021f0bda9a71caa974e7ea2db1ee25fa9de86cb2d7de26f360e: Status 404 returned error can't find the container with id 4b3f6f5e2076b021f0bda9a71caa974e7ea2db1ee25fa9de86cb2d7de26f360e
Jul 20 10:31:11.767206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.767185    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:31:11.930236 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:11.930139    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:12.068384 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:12.068315    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"]
Jul 20 10:31:12.070515 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:31:12.070486    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31bb17e_3044_4925_8c5e_b2ec82810be4.slice/crio-5c5c529e38fde6e8fb39757d74690d3da5daca469322cf83244130432861f25f WatchSource:0}: Error finding container 5c5c529e38fde6e8fb39757d74690d3da5daca469322cf83244130432861f25f: Status 404 returned error can't find the container with id 5c5c529e38fde6e8fb39757d74690d3da5daca469322cf83244130432861f25f
Jul 20 10:31:12.114537 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:12.114508    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" event={"ID":"a31bb17e-3044-4925-8c5e-b2ec82810be4","Type":"ContainerStarted","Data":"5c5c529e38fde6e8fb39757d74690d3da5daca469322cf83244130432861f25f"}
Jul 20 10:31:12.115606 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:12.115587    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" event={"ID":"352b76fd-c536-469a-a96b-0b7df702b444","Type":"ContainerStarted","Data":"4b3f6f5e2076b021f0bda9a71caa974e7ea2db1ee25fa9de86cb2d7de26f360e"}
Jul 20 10:31:13.123057 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:13.122972    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" event={"ID":"a31bb17e-3044-4925-8c5e-b2ec82810be4","Type":"ContainerStarted","Data":"401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3"}
Jul 20 10:31:13.147622 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:13.147550    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podStartSLOduration=10.147535777 podStartE2EDuration="10.147535777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:31:13.14490331 +0000 UTC m=+2053.158005220" watchObservedRunningTime="2026-07-20 10:31:13.147535777 +0000 UTC m=+2053.160637673"
Jul 20 10:31:14.129672 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:14.129632    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" event={"ID":"352b76fd-c536-469a-a96b-0b7df702b444","Type":"ContainerStarted","Data":"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9"}
Jul 20 10:31:15.137720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:15.137679    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" event={"ID":"352b76fd-c536-469a-a96b-0b7df702b444","Type":"ContainerStarted","Data":"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7"}
Jul 20 10:31:15.138151 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:15.137823    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:15.166328 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:15.166262    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podStartSLOduration=10.570306378 podStartE2EDuration="12.166242377s" totalImagesPullingTime="1.595935999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:31:15.163164713 +0000 UTC m=+2055.176266632" watchObservedRunningTime="2026-07-20 10:31:15.166242377 +0000 UTC m=+2055.179344275"
Jul 20 10:31:21.620521 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:21.620476    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:21.621114 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:21.620576    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:21.622027 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:21.621994    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:31:21.930884 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:21.930752    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:21.930884 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:21.930835    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:31:21.932442 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:21.932409    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:31:31.620879 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:31.620828    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:31:31.636841 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:31.636809    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:31:31.930579 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:31.930446    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:31:41.621002 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:41.620940    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:31:41.931392 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:41.931285    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:31:51.621068 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:51.621016    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:31:51.931318 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:31:51.931211    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:32:00.529011 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:00.528979    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:32:00.538203 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:00.538171    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:32:01.620938 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:01.620885    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:32:01.930869 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:01.930761    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:32:11.620743 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:11.620688    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:32:11.931014 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:11.930909    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:32:21.620690 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:21.620638    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:32:21.930926 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:21.930829    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:32:31.620361 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:31.620110    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:32:31.930587 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:31.930484    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:32:41.620836 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:41.620776    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:32:41.930672 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:41.930567    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:32:51.620695 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:51.620638    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:32:51.930982 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:32:51.930867    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:33:01.620706 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:01.620655    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:33:01.931480 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:01.931383    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:33:11.621014 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:11.620965    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:33:11.931064 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:11.930966    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:33:21.620544 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:21.620489    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:33:21.930482 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:21.930386    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:33:31.620389 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:31.620333    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:33:31.930786 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:31.930680    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:33:41.620682 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:41.620632    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8001/health\": dial tcp 10.134.0.51:8001: connect: connection refused"
Jul 20 10:33:41.930943 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:41.930850    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 20 10:33:51.630016 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:51.629979    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:33:51.641658 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:51.641631    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:33:51.942468 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:51.942389    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:33:51.950088 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:51.950058    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:33:59.619823 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:59.619790    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"]
Jul 20 10:33:59.624536 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:33:59.624489    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"]
Jul 20 10:34:00.667135 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:00.667100    2572 scope.go:117] "RemoveContainer" containerID="8a0b6d01ab70bfde81a41671fb3dbd00c2e35b7abecea3f177441f54c5ab9bf0"
Jul 20 10:34:14.646047 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:14.645984    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main" containerID="cri-o://401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3" gracePeriod=45
Jul 20 10:34:14.647055 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:14.647022    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main" containerID="cri-o://0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7" gracePeriod=45
Jul 20 10:34:15.463133 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.463087    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="llm-d-routing-sidecar" containerID="cri-o://b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9" gracePeriod=45
Jul 20 10:34:15.593429 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.593405    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:34:15.617477 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.617442    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-dshm\") pod \"a31bb17e-3044-4925-8c5e-b2ec82810be4\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") "
Jul 20 10:34:15.617676 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.617490    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-model-cache\") pod \"a31bb17e-3044-4925-8c5e-b2ec82810be4\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") "
Jul 20 10:34:15.617676 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.617627    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"a31bb17e-3044-4925-8c5e-b2ec82810be4\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") "
Jul 20 10:34:15.617676 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.617662    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-tmp-dir\") pod \"a31bb17e-3044-4925-8c5e-b2ec82810be4\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") "
Jul 20 10:34:15.617857 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.617708    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsrgf\" (UniqueName: \"kubernetes.io/projected/a31bb17e-3044-4925-8c5e-b2ec82810be4-kube-api-access-dsrgf\") pod \"a31bb17e-3044-4925-8c5e-b2ec82810be4\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") "
Jul 20 10:34:15.617857 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.617736    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31bb17e-3044-4925-8c5e-b2ec82810be4-tls-certs\") pod \"a31bb17e-3044-4925-8c5e-b2ec82810be4\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") "
Jul 20 10:34:15.617857 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.617788    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-home\") pod \"a31bb17e-3044-4925-8c5e-b2ec82810be4\" (UID: \"a31bb17e-3044-4925-8c5e-b2ec82810be4\") "
Jul 20 10:34:15.617857 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.617786    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-model-cache" (OuterVolumeSpecName: "model-cache") pod "a31bb17e-3044-4925-8c5e-b2ec82810be4" (UID: "a31bb17e-3044-4925-8c5e-b2ec82810be4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.618072 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.618008    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.618644 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.618616    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-home" (OuterVolumeSpecName: "home") pod "a31bb17e-3044-4925-8c5e-b2ec82810be4" (UID: "a31bb17e-3044-4925-8c5e-b2ec82810be4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.620732 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.620699    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31bb17e-3044-4925-8c5e-b2ec82810be4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a31bb17e-3044-4925-8c5e-b2ec82810be4" (UID: "a31bb17e-3044-4925-8c5e-b2ec82810be4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:34:15.620844 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.620737    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31bb17e-3044-4925-8c5e-b2ec82810be4-kube-api-access-dsrgf" (OuterVolumeSpecName: "kube-api-access-dsrgf") pod "a31bb17e-3044-4925-8c5e-b2ec82810be4" (UID: "a31bb17e-3044-4925-8c5e-b2ec82810be4"). InnerVolumeSpecName "kube-api-access-dsrgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:34:15.620987 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.620962    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed" (OuterVolumeSpecName: "kserve-pvc-source") pod "a31bb17e-3044-4925-8c5e-b2ec82810be4" (UID: "a31bb17e-3044-4925-8c5e-b2ec82810be4"). InnerVolumeSpecName "pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 20 10:34:15.621167 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.621138    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-dshm" (OuterVolumeSpecName: "dshm") pod "a31bb17e-3044-4925-8c5e-b2ec82810be4" (UID: "a31bb17e-3044-4925-8c5e-b2ec82810be4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.636507 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.636438    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a31bb17e-3044-4925-8c5e-b2ec82810be4" (UID: "a31bb17e-3044-4925-8c5e-b2ec82810be4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.718874 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.718849    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.718874 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.718873    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.719211 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.718883    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsrgf\" (UniqueName: \"kubernetes.io/projected/a31bb17e-3044-4925-8c5e-b2ec82810be4-kube-api-access-dsrgf\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.719211 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.718891    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31bb17e-3044-4925-8c5e-b2ec82810be4-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.719211 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.718899    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31bb17e-3044-4925-8c5e-b2ec82810be4-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.721535 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.721520    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:34:15.819436 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.819404    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4qcs\" (UniqueName: \"kubernetes.io/projected/352b76fd-c536-469a-a96b-0b7df702b444-kube-api-access-z4qcs\") pod \"352b76fd-c536-469a-a96b-0b7df702b444\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") "
Jul 20 10:34:15.819610 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.819466    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-tmp-dir\") pod \"352b76fd-c536-469a-a96b-0b7df702b444\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") "
Jul 20 10:34:15.819610 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.819499    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-home\") pod \"352b76fd-c536-469a-a96b-0b7df702b444\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") "
Jul 20 10:34:15.819709 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.819634    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") pod \"352b76fd-c536-469a-a96b-0b7df702b444\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") "
Jul 20 10:34:15.819709 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.819681    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-dshm\") pod \"352b76fd-c536-469a-a96b-0b7df702b444\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") "
Jul 20 10:34:15.819800 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.819710    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352b76fd-c536-469a-a96b-0b7df702b444-tls-certs\") pod \"352b76fd-c536-469a-a96b-0b7df702b444\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") "
Jul 20 10:34:15.819800 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.819737    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-model-cache\") pod \"352b76fd-c536-469a-a96b-0b7df702b444\" (UID: \"352b76fd-c536-469a-a96b-0b7df702b444\") "
Jul 20 10:34:15.820189 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.820162    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-model-cache" (OuterVolumeSpecName: "model-cache") pod "352b76fd-c536-469a-a96b-0b7df702b444" (UID: "352b76fd-c536-469a-a96b-0b7df702b444"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.820327 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.820222    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-home" (OuterVolumeSpecName: "home") pod "352b76fd-c536-469a-a96b-0b7df702b444" (UID: "352b76fd-c536-469a-a96b-0b7df702b444"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.822023 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.821983    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-dshm" (OuterVolumeSpecName: "dshm") pod "352b76fd-c536-469a-a96b-0b7df702b444" (UID: "352b76fd-c536-469a-a96b-0b7df702b444"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.822180 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.822147    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352b76fd-c536-469a-a96b-0b7df702b444-kube-api-access-z4qcs" (OuterVolumeSpecName: "kube-api-access-z4qcs") pod "352b76fd-c536-469a-a96b-0b7df702b444" (UID: "352b76fd-c536-469a-a96b-0b7df702b444"). InnerVolumeSpecName "kube-api-access-z4qcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:34:15.822656 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.822629    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352b76fd-c536-469a-a96b-0b7df702b444-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "352b76fd-c536-469a-a96b-0b7df702b444" (UID: "352b76fd-c536-469a-a96b-0b7df702b444"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:34:15.822656 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.822640    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed" (OuterVolumeSpecName: "kserve-pvc-source") pod "352b76fd-c536-469a-a96b-0b7df702b444" (UID: "352b76fd-c536-469a-a96b-0b7df702b444"). InnerVolumeSpecName "pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 20 10:34:15.832787 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.832762    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "352b76fd-c536-469a-a96b-0b7df702b444" (UID: "352b76fd-c536-469a-a96b-0b7df702b444"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:34:15.920593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.920462    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qcs\" (UniqueName: \"kubernetes.io/projected/352b76fd-c536-469a-a96b-0b7df702b444-kube-api-access-z4qcs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.920593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.920500    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.920593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.920514    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.920593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.920549    2572 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") on node \"ip-10-0-138-196.ec2.internal\" "
Jul 20 10:34:15.920593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.920585    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.920593 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.920599    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352b76fd-c536-469a-a96b-0b7df702b444-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.920901 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.920611    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352b76fd-c536-469a-a96b-0b7df702b444-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:15.947169 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.947150    2572 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed") on node "ip-10-0-138-196.ec2.internal"
Jul 20 10:34:15.996108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.996077    2572 generic.go:356] "Generic (PLEG): container finished" podID="352b76fd-c536-469a-a96b-0b7df702b444" containerID="0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7" exitCode=0
Jul 20 10:34:15.996108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.996106    2572 generic.go:356] "Generic (PLEG): container finished" podID="352b76fd-c536-469a-a96b-0b7df702b444" containerID="b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9" exitCode=0
Jul 20 10:34:15.996285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.996135    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"
Jul 20 10:34:15.996285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.996163    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" event={"ID":"352b76fd-c536-469a-a96b-0b7df702b444","Type":"ContainerDied","Data":"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7"}
Jul 20 10:34:15.996285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.996222    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" event={"ID":"352b76fd-c536-469a-a96b-0b7df702b444","Type":"ContainerDied","Data":"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9"}
Jul 20 10:34:15.996285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.996241    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6" event={"ID":"352b76fd-c536-469a-a96b-0b7df702b444","Type":"ContainerDied","Data":"4b3f6f5e2076b021f0bda9a71caa974e7ea2db1ee25fa9de86cb2d7de26f360e"}
Jul 20 10:34:15.996285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.996263    2572 scope.go:117] "RemoveContainer" containerID="0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7"
Jul 20 10:34:15.997584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.997548    2572 generic.go:356] "Generic (PLEG): container finished" podID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerID="401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3" exitCode=0
Jul 20 10:34:15.997684 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.997597    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" event={"ID":"a31bb17e-3044-4925-8c5e-b2ec82810be4","Type":"ContainerDied","Data":"401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3"}
Jul 20 10:34:15.997684 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.997620    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"
Jul 20 10:34:15.997684 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:15.997629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n" event={"ID":"a31bb17e-3044-4925-8c5e-b2ec82810be4","Type":"ContainerDied","Data":"5c5c529e38fde6e8fb39757d74690d3da5daca469322cf83244130432861f25f"}
Jul 20 10:34:16.005807 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.005790    2572 scope.go:117] "RemoveContainer" containerID="b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9"
Jul 20 10:34:16.013503 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.013489    2572 scope.go:117] "RemoveContainer" containerID="0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7"
Jul 20 10:34:16.013792 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:34:16.013773    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7\": container with ID starting with 0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7 not found: ID does not exist" containerID="0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7"
Jul 20 10:34:16.013847 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.013799    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7"} err="failed to get container status \"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7\": rpc error: code = NotFound desc = could not find container \"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7\": container with ID starting with 0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7 not found: ID does not exist"
Jul 20 10:34:16.013847 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.013815    2572 scope.go:117] "RemoveContainer" containerID="b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9"
Jul 20 10:34:16.014045 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:34:16.014028    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9\": container with ID starting with b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9 not found: ID does not exist" containerID="b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9"
Jul 20 10:34:16.014085 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.014051    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9"} err="failed to get container status \"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9\": rpc error: code = NotFound desc = could not find container \"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9\": container with ID starting with b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9 not found: ID does not exist"
Jul 20 10:34:16.014085 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.014066    2572 scope.go:117] "RemoveContainer" containerID="0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7"
Jul 20 10:34:16.014302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.014284    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7"} err="failed to get container status \"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7\": rpc error: code = NotFound desc = could not find container \"0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7\": container with ID starting with 0b8e4e6f760927327012484fa97756913e490b531a7c34d763af9f04e0b868c7 not found: ID does not exist"
Jul 20 10:34:16.014302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.014302    2572 scope.go:117] "RemoveContainer" containerID="b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9"
Jul 20 10:34:16.014476 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.014461    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9"} err="failed to get container status \"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9\": rpc error: code = NotFound desc = could not find container \"b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9\": container with ID starting with b07b7d3ea7f025b26551ce815d6b9437c5fd8eb5a3618ca449a16a3ba0e4abb9 not found: ID does not exist"
Jul 20 10:34:16.014516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.014476    2572 scope.go:117] "RemoveContainer" containerID="401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3"
Jul 20 10:34:16.021090 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.021067    2572 reconciler_common.go:299] "Volume detached for volume \"pvc-aaab91f0-341f-403b-8e7a-0bdb9b281b4b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-041627ecd085904ed\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"csi-96f8f215107b6e036516bdbf320e5d4c7f0cf98c203f1213a91942cce6bf9133\""
Jul 20 10:34:16.022022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.021999    2572 scope.go:117] "RemoveContainer" containerID="401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3"
Jul 20 10:34:16.022295 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:34:16.022272    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3\": container with ID starting with 401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3 not found: ID does not exist" containerID="401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3"
Jul 20 10:34:16.022399 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.022299    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3"} err="failed to get container status \"401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3\": rpc error: code = NotFound desc = could not find container \"401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3\": container with ID starting with 401202ca5758547f17d69d1394a971d1a0eacd259cdc7df531c0268b4da8d9a3 not found: ID does not exist"
Jul 20 10:34:16.022834 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.022816    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"]
Jul 20 10:34:16.026755 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.026735    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-66b5bb5cc4frvj6"]
Jul 20 10:34:16.040677 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.040651    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"]
Jul 20 10:34:16.044852 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.044822    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-df65l8n"]
Jul 20 10:34:16.520533 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.520499    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="352b76fd-c536-469a-a96b-0b7df702b444" path="/var/lib/kubelet/pods/352b76fd-c536-469a-a96b-0b7df702b444/volumes"
Jul 20 10:34:16.520990 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:16.520976    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" path="/var/lib/kubelet/pods/a31bb17e-3044-4925-8c5e-b2ec82810be4/volumes"
Jul 20 10:34:22.976264 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976175    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"]
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976465    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main"
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976475    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main"
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976485    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main"
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976490    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main"
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976497    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="llm-d-routing-sidecar"
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976503    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="llm-d-routing-sidecar"
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976579    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="llm-d-routing-sidecar"
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976588    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="352b76fd-c536-469a-a96b-0b7df702b444" containerName="main"
Jul 20 10:34:22.976638 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.976598    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a31bb17e-3044-4925-8c5e-b2ec82810be4" containerName="main"
Jul 20 10:34:22.981198 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.981175    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:22.984083 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.984062    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 20 10:34:22.984200 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.984082    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-s5hvh\""
Jul 20 10:34:22.985293 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.985277    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 20 10:34:22.987741 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:22.987720    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"]
Jul 20 10:34:23.075930 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:23.075901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6qd\" (UniqueName: \"kubernetes.io/projected/e03b09f0-2ba7-4dd7-9d48-0818ffa18033-kube-api-access-8b6qd\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:23.176328 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:23.176294    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6qd\" (UniqueName: \"kubernetes.io/projected/e03b09f0-2ba7-4dd7-9d48-0818ffa18033-kube-api-access-8b6qd\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:23.185305 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:23.185274    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6qd\" (UniqueName: \"kubernetes.io/projected/e03b09f0-2ba7-4dd7-9d48-0818ffa18033-kube-api-access-8b6qd\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:24.691998 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:24.691962    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"]
Jul 20 10:34:24.695710 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:24.695684    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-gqlt6"]
Jul 20 10:34:25.496314 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:25.496272    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:25.503539 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:25.503515    2572 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:25.596772 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:25.596735    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:25.596923 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:25.596879    2572 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:25.600205 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:25.600188    2572 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") DevicePath \"csi-52142cb8d77ff7ede58a62cde42b286fed755369dd19ccc158de8d922a9c65e2\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:25.797741 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:25.797703    2572 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/bb4159b2622cc0182d31261450f277bc65a5e677069868c28b18a44e5d4aeddc/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:25.811395 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:25.811368    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"e2e-pvc-model-download-m9fbp\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:25.990721 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:25.990685    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:26.123193 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:26.123037    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"]
Jul 20 10:34:26.125889 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:34:26.125855    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode03b09f0_2ba7_4dd7_9d48_0818ffa18033.slice/crio-de66b48a72d0810987ec56ab3bbc484c0d170f589815996c8889e6308d81dfc6 WatchSource:0}: Error finding container de66b48a72d0810987ec56ab3bbc484c0d170f589815996c8889e6308d81dfc6: Status 404 returned error can't find the container with id de66b48a72d0810987ec56ab3bbc484c0d170f589815996c8889e6308d81dfc6
Jul 20 10:34:26.519351 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:26.519315    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f0f3e23-1b34-4eb9-846d-4082355c5886" path="/var/lib/kubelet/pods/6f0f3e23-1b34-4eb9-846d-4082355c5886/volumes"
Jul 20 10:34:27.054116 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:27.054076    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp" event={"ID":"e03b09f0-2ba7-4dd7-9d48-0818ffa18033","Type":"ContainerStarted","Data":"aac56d1e6f3c3d8eed8619447220e531591e0b8e7795bc880606970442b9a53f"}
Jul 20 10:34:27.054646 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:27.054620    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp" event={"ID":"e03b09f0-2ba7-4dd7-9d48-0818ffa18033","Type":"ContainerStarted","Data":"de66b48a72d0810987ec56ab3bbc484c0d170f589815996c8889e6308d81dfc6"}
Jul 20 10:34:27.073217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:27.073133    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp" podStartSLOduration=8.073101778 podStartE2EDuration="8.073101778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:34:27.07086289 +0000 UTC m=+2247.083964788" watchObservedRunningTime="2026-07-20 10:34:27.073101778 +0000 UTC m=+2247.086203669"
Jul 20 10:34:33.084005 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:33.083969    2572 generic.go:356] "Generic (PLEG): container finished" podID="e03b09f0-2ba7-4dd7-9d48-0818ffa18033" containerID="aac56d1e6f3c3d8eed8619447220e531591e0b8e7795bc880606970442b9a53f" exitCode=0
Jul 20 10:34:33.084445 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:33.084044    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp" event={"ID":"e03b09f0-2ba7-4dd7-9d48-0818ffa18033","Type":"ContainerDied","Data":"aac56d1e6f3c3d8eed8619447220e531591e0b8e7795bc880606970442b9a53f"}
Jul 20 10:34:34.234396 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.234367    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:34.371356 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.371286    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") "
Jul 20 10:34:34.371356 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.371349    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b6qd\" (UniqueName: \"kubernetes.io/projected/e03b09f0-2ba7-4dd7-9d48-0818ffa18033-kube-api-access-8b6qd\") pod \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\" (UID: \"e03b09f0-2ba7-4dd7-9d48-0818ffa18033\") "
Jul 20 10:34:34.373740 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.373708    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03b09f0-2ba7-4dd7-9d48-0818ffa18033-kube-api-access-8b6qd" (OuterVolumeSpecName: "kube-api-access-8b6qd") pod "e03b09f0-2ba7-4dd7-9d48-0818ffa18033" (UID: "e03b09f0-2ba7-4dd7-9d48-0818ffa18033"). InnerVolumeSpecName "kube-api-access-8b6qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:34:34.374168 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.374145    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571" (OuterVolumeSpecName: "model-storage") pod "e03b09f0-2ba7-4dd7-9d48-0818ffa18033" (UID: "e03b09f0-2ba7-4dd7-9d48-0818ffa18033"). InnerVolumeSpecName "pvc-c758418e-53f5-477b-bdd9-aecc24c01911". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 20 10:34:34.472503 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.472469    2572 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") on node \"ip-10-0-138-196.ec2.internal\" "
Jul 20 10:34:34.472503 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.472500    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6qd\" (UniqueName: \"kubernetes.io/projected/e03b09f0-2ba7-4dd7-9d48-0818ffa18033-kube-api-access-8b6qd\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:34:34.593818 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.593783    2572 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-c758418e-53f5-477b-bdd9-aecc24c01911" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571") on node "ip-10-0-138-196.ec2.internal"
Jul 20 10:34:34.674193 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:34.674107    2572 reconciler_common.go:299] "Volume detached for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"csi-52142cb8d77ff7ede58a62cde42b286fed755369dd19ccc158de8d922a9c65e2\""
Jul 20 10:34:35.092775 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:35.092733    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp" event={"ID":"e03b09f0-2ba7-4dd7-9d48-0818ffa18033","Type":"ContainerDied","Data":"de66b48a72d0810987ec56ab3bbc484c0d170f589815996c8889e6308d81dfc6"}
Jul 20 10:34:35.092775 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:35.092775    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de66b48a72d0810987ec56ab3bbc484c0d170f589815996c8889e6308d81dfc6"
Jul 20 10:34:35.092775 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:35.092753    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"
Jul 20 10:34:42.599084 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.599046    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 20 10:34:42.599476 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.599339    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03b09f0-2ba7-4dd7-9d48-0818ffa18033" containerName="storage-initializer"
Jul 20 10:34:42.599476 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.599349    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03b09f0-2ba7-4dd7-9d48-0818ffa18033" containerName="storage-initializer"
Jul 20 10:34:42.599476 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.599394    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03b09f0-2ba7-4dd7-9d48-0818ffa18033" containerName="storage-initializer"
Jul 20 10:34:42.602224 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.602206    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.607129 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.607106    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-s5hvh\""
Jul 20 10:34:42.607129 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.607117    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-krjqn\""
Jul 20 10:34:42.607347 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.607234    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 20 10:34:42.607347 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.607326    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 20 10:34:42.618365 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.618347    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 20 10:34:42.632339 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.632315    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.632479 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.632441    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.632598 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.632579    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.632680 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.632622    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.632737 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.632681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.634598 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.634574    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 20 10:34:42.726891 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.726857    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 20 10:34:42.730083 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.730065    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.733907 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.733888    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.734032 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.733914    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.734032 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.733952    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.734032 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.733976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9d42\" (UniqueName: \"kubernetes.io/projected/f7b021d8-977a-406c-bf43-a70d0e7ac03b-kube-api-access-m9d42\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.734032 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.733999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.734032 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.734019    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.734301 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.734281    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.734403 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.734383    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.734459 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.734424    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.736174 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.736151    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.736509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.736489    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.750712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.750687    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 20 10:34:42.835090 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.835054    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.835292 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.835113    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.835292 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.835194    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.835292 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.835274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8f6\" (UniqueName: \"kubernetes.io/projected/ddfe9255-2b02-43c1-b532-20641d7e3848-kube-api-access-9r8f6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.835461 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.835336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9d42\" (UniqueName: \"kubernetes.io/projected/f7b021d8-977a-406c-bf43-a70d0e7ac03b-kube-api-access-m9d42\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.835461 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.835371    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.835461 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.835408    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.859057 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.858990    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9d42\" (UniqueName: \"kubernetes.io/projected/f7b021d8-977a-406c-bf43-a70d0e7ac03b-kube-api-access-m9d42\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:42.935843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.935794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.935843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.935835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.936097 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.935885    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8f6\" (UniqueName: \"kubernetes.io/projected/ddfe9255-2b02-43c1-b532-20641d7e3848-kube-api-access-9r8f6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.936097 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.935939    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.936097 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.935974    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.936097 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.936016    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.936315 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.936285    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.936374 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.936333    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.936434 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.936393    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.938193 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.938162    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.938445 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.938423    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:42.949349 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:42.949325    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8f6\" (UniqueName: \"kubernetes.io/projected/ddfe9255-2b02-43c1-b532-20641d7e3848-kube-api-access-9r8f6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:45.359670 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.359631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:45.378536 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.378503    2572 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:45.460897 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.460862    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:45.461074 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.461002    2572 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:45.467105 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.467074    2572 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") DevicePath \"csi-52142cb8d77ff7ede58a62cde42b286fed755369dd19ccc158de8d922a9c65e2\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:45.559205 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.559174    2572 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/bb4159b2622cc0182d31261450f277bc65a5e677069868c28b18a44e5d4aeddc/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:45.577905 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.577879    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:45.612382 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.612312    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:45.662793 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.662763    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:45.662960 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.662882    2572 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") DevicePath \"csi-52142cb8d77ff7ede58a62cde42b286fed755369dd19ccc158de8d922a9c65e2\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:45.672621 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.672582    2572 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") DevicePath \"csi-52142cb8d77ff7ede58a62cde42b286fed755369dd19ccc158de8d922a9c65e2\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:45.691309 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.691279    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:45.746999 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.746966    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:34:45.791665 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.791642    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 20 10:34:45.794835 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:34:45.794807    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b021d8_977a_406c_bf43_a70d0e7ac03b.slice/crio-ec70a66f675c56dec674db8f3efb403e3b4f2c8f13e99acef1cb2fb8936b3991 WatchSource:0}: Error finding container ec70a66f675c56dec674db8f3efb403e3b4f2c8f13e99acef1cb2fb8936b3991: Status 404 returned error can't find the container with id ec70a66f675c56dec674db8f3efb403e3b4f2c8f13e99acef1cb2fb8936b3991
Jul 20 10:34:45.903747 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:45.903722    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 20 10:34:45.905635 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:34:45.905608    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfe9255_2b02_43c1_b532_20641d7e3848.slice/crio-fc7366ad11f8323e7e3ec41f8256a18c0003c8dddb7753f88a665f4e3b80643b WatchSource:0}: Error finding container fc7366ad11f8323e7e3ec41f8256a18c0003c8dddb7753f88a665f4e3b80643b: Status 404 returned error can't find the container with id fc7366ad11f8323e7e3ec41f8256a18c0003c8dddb7753f88a665f4e3b80643b
Jul 20 10:34:46.143319 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:46.143224    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"f7b021d8-977a-406c-bf43-a70d0e7ac03b","Type":"ContainerStarted","Data":"a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73"}
Jul 20 10:34:46.143319 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:46.143285    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"f7b021d8-977a-406c-bf43-a70d0e7ac03b","Type":"ContainerStarted","Data":"ec70a66f675c56dec674db8f3efb403e3b4f2c8f13e99acef1cb2fb8936b3991"}
Jul 20 10:34:46.144971 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:46.144941    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"ddfe9255-2b02-43c1-b532-20641d7e3848","Type":"ContainerStarted","Data":"12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a"}
Jul 20 10:34:46.145107 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:46.144996    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"ddfe9255-2b02-43c1-b532-20641d7e3848","Type":"ContainerStarted","Data":"fc7366ad11f8323e7e3ec41f8256a18c0003c8dddb7753f88a665f4e3b80643b"}
Jul 20 10:34:46.164323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:46.164272    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=4.164257975 podStartE2EDuration="4.164257975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:34:46.163444832 +0000 UTC m=+2266.176546740" watchObservedRunningTime="2026-07-20 10:34:46.164257975 +0000 UTC m=+2266.177359871"
Jul 20 10:34:46.183072 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:46.183011    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=4.182990624 podStartE2EDuration="4.182990624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:34:46.182211392 +0000 UTC m=+2266.195313288" watchObservedRunningTime="2026-07-20 10:34:46.182990624 +0000 UTC m=+2266.196092524"
Jul 20 10:34:46.613044 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:46.613011    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:55.613401 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:55.613356    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:34:55.614842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:34:55.614808    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:35:05.613396 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:35:05.613341    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:35:15.613152 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:35:15.613095    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:35:25.613776 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:35:25.613728    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:35:35.613234 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:35:35.613176    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:35:45.613752 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:35:45.613705    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:35:55.613112 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:35:55.613015    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:36:05.613589 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:36:05.613531    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:36:15.613599 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:36:15.613528    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:36:25.613068 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:36:25.613012    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:36:35.613190 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:36:35.613130    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:36:45.613867 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:36:45.613815    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:36:55.613298 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:36:55.613254    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 20 10:37:00.554291 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:37:00.554259    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:37:00.568405 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:37:00.568384    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:37:00.712089 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:37:00.712051    2572 scope.go:117] "RemoveContainer" containerID="07b0d938103ef1e3d79dc9afed9eba3afca5d89c49b41e145cd0c6d24d9eb1b9"
Jul 20 10:37:05.622697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:37:05.622659    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:37:05.630143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:37:05.630117    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:39:22.757838 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:22.757801    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 20 10:39:29.514832 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:29.514800    2572 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-krjqn\" not found"
Jul 20 10:39:29.551871 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:29.551845    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:29.551970 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:29.551934    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs podName:ddfe9255-2b02-43c1-b532-20641d7e3848 nodeName:}" failed. No retries permitted until 2026-07-20 10:39:30.051916954 +0000 UTC m=+2550.065018828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:30.056496 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:30.056463    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:30.056712 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:30.056529    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs podName:ddfe9255-2b02-43c1-b532-20641d7e3848 nodeName:}" failed. No retries permitted until 2026-07-20 10:39:31.056514742 +0000 UTC m=+2551.069616616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:31.065039 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:31.065011    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:31.065442 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:31.065093    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs podName:ddfe9255-2b02-43c1-b532-20641d7e3848 nodeName:}" failed. No retries permitted until 2026-07-20 10:39:33.065070958 +0000 UTC m=+2553.078172832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:33.085234 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:33.085197    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:33.085665 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:33.085275    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs podName:ddfe9255-2b02-43c1-b532-20641d7e3848 nodeName:}" failed. No retries permitted until 2026-07-20 10:39:37.085260602 +0000 UTC m=+2557.098362476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:37.127054 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:37.127022    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:37.127440 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:37.127101    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs podName:ddfe9255-2b02-43c1-b532-20641d7e3848 nodeName:}" failed. No retries permitted until 2026-07-20 10:39:45.127085858 +0000 UTC m=+2565.140187732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:37.781513 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:37.781465    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main" containerID="cri-o://a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73" gracePeriod=45
Jul 20 10:39:38.455405 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.455382    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:39:38.460669 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.460643    2572 generic.go:356] "Generic (PLEG): container finished" podID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerID="a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73" exitCode=0
Jul 20 10:39:38.460777 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.460679    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 20 10:39:38.460777 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.460720    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"f7b021d8-977a-406c-bf43-a70d0e7ac03b","Type":"ContainerDied","Data":"a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73"}
Jul 20 10:39:38.460777 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.460764    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"f7b021d8-977a-406c-bf43-a70d0e7ac03b","Type":"ContainerDied","Data":"ec70a66f675c56dec674db8f3efb403e3b4f2c8f13e99acef1cb2fb8936b3991"}
Jul 20 10:39:38.460889 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.460795    2572 scope.go:117] "RemoveContainer" containerID="a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73"
Jul 20 10:39:38.468531 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.468514    2572 scope.go:117] "RemoveContainer" containerID="a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73"
Jul 20 10:39:38.468798 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:38.468780    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73\": container with ID starting with a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73 not found: ID does not exist" containerID="a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73"
Jul 20 10:39:38.468852 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.468812    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73"} err="failed to get container status \"a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73\": rpc error: code = NotFound desc = could not find container \"a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73\": container with ID starting with a11601d5f06ab287fb2b9a1dd612c1eab16604b0e3400afaa9f24e02c86a7b73 not found: ID does not exist"
Jul 20 10:39:38.539837 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.539772    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9d42\" (UniqueName: \"kubernetes.io/projected/f7b021d8-977a-406c-bf43-a70d0e7ac03b-kube-api-access-m9d42\") pod \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") "
Jul 20 10:39:38.539837 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.539814    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tls-certs\") pod \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") "
Jul 20 10:39:38.540022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.539851    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-model-cache\") pod \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") "
Jul 20 10:39:38.540022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.539957    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-dshm\") pod \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") "
Jul 20 10:39:38.540111 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.540049    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") "
Jul 20 10:39:38.540111 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.540092    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7b021d8-977a-406c-bf43-a70d0e7ac03b" (UID: "f7b021d8-977a-406c-bf43-a70d0e7ac03b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:38.540203 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.540116    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-home\") pod \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") "
Jul 20 10:39:38.540203 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.540145    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tmp-dir\") pod \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\" (UID: \"f7b021d8-977a-406c-bf43-a70d0e7ac03b\") "
Jul 20 10:39:38.540401 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.540378    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:38.540842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.540804    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-home" (OuterVolumeSpecName: "home") pod "f7b021d8-977a-406c-bf43-a70d0e7ac03b" (UID: "f7b021d8-977a-406c-bf43-a70d0e7ac03b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:38.542305 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.542276    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-dshm" (OuterVolumeSpecName: "dshm") pod "f7b021d8-977a-406c-bf43-a70d0e7ac03b" (UID: "f7b021d8-977a-406c-bf43-a70d0e7ac03b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:38.542429 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.542308    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7b021d8-977a-406c-bf43-a70d0e7ac03b" (UID: "f7b021d8-977a-406c-bf43-a70d0e7ac03b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:39:38.542429 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.542385    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b021d8-977a-406c-bf43-a70d0e7ac03b-kube-api-access-m9d42" (OuterVolumeSpecName: "kube-api-access-m9d42") pod "f7b021d8-977a-406c-bf43-a70d0e7ac03b" (UID: "f7b021d8-977a-406c-bf43-a70d0e7ac03b"). InnerVolumeSpecName "kube-api-access-m9d42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:39:38.543170 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.543149    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571" (OuterVolumeSpecName: "kserve-pvc-source") pod "f7b021d8-977a-406c-bf43-a70d0e7ac03b" (UID: "f7b021d8-977a-406c-bf43-a70d0e7ac03b"). InnerVolumeSpecName "pvc-c758418e-53f5-477b-bdd9-aecc24c01911". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 20 10:39:38.551721 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.551700    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7b021d8-977a-406c-bf43-a70d0e7ac03b" (UID: "f7b021d8-977a-406c-bf43-a70d0e7ac03b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:38.641091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.641055    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:38.641091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.641094    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:38.641279 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.641107    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:38.641279 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.641123    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9d42\" (UniqueName: \"kubernetes.io/projected/f7b021d8-977a-406c-bf43-a70d0e7ac03b-kube-api-access-m9d42\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:38.641279 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.641135    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b021d8-977a-406c-bf43-a70d0e7ac03b-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:38.783874 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.783842    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 20 10:39:38.788609 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.788585    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 20 10:39:38.817134 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:38.817032    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 20 10:39:40.519227 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:40.519193    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" path="/var/lib/kubelet/pods/f7b021d8-977a-406c-bf43-a70d0e7ac03b/volumes"
Jul 20 10:39:45.198812 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:45.198776    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:45.199208 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:45.198856    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs podName:ddfe9255-2b02-43c1-b532-20641d7e3848 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:01.198839892 +0000 UTC m=+2581.211941766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 20 10:39:53.840148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:53.840099    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="ddfe9255-2b02-43c1-b532-20641d7e3848" containerName="main" containerID="cri-o://12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a" gracePeriod=45
Jul 20 10:39:54.508961 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.508939    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:39:54.527281 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.527247    2572 generic.go:356] "Generic (PLEG): container finished" podID="ddfe9255-2b02-43c1-b532-20641d7e3848" containerID="12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a" exitCode=0
Jul 20 10:39:54.527398 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.527290    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 20 10:39:54.527398 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.527330    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"ddfe9255-2b02-43c1-b532-20641d7e3848","Type":"ContainerDied","Data":"12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a"}
Jul 20 10:39:54.527398 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.527384    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"ddfe9255-2b02-43c1-b532-20641d7e3848","Type":"ContainerDied","Data":"fc7366ad11f8323e7e3ec41f8256a18c0003c8dddb7753f88a665f4e3b80643b"}
Jul 20 10:39:54.527508 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.527408    2572 scope.go:117] "RemoveContainer" containerID="12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a"
Jul 20 10:39:54.535516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.535499    2572 scope.go:117] "RemoveContainer" containerID="12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a"
Jul 20 10:39:54.535843 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:39:54.535824    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a\": container with ID starting with 12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a not found: ID does not exist" containerID="12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a"
Jul 20 10:39:54.535944 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.535851    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a"} err="failed to get container status \"12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a\": rpc error: code = NotFound desc = could not find container \"12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a\": container with ID starting with 12583d8b17325fb5a60df9c391e151dee19f3d20484ea4ce7da77d3455cfcc3a not found: ID does not exist"
Jul 20 10:39:54.584033 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584005    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-tmp-dir\") pod \"ddfe9255-2b02-43c1-b532-20641d7e3848\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") "
Jul 20 10:39:54.584195 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584048    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-home\") pod \"ddfe9255-2b02-43c1-b532-20641d7e3848\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") "
Jul 20 10:39:54.584195 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584073    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-dshm\") pod \"ddfe9255-2b02-43c1-b532-20641d7e3848\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") "
Jul 20 10:39:54.584195 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584109    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r8f6\" (UniqueName: \"kubernetes.io/projected/ddfe9255-2b02-43c1-b532-20641d7e3848-kube-api-access-9r8f6\") pod \"ddfe9255-2b02-43c1-b532-20641d7e3848\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") "
Jul 20 10:39:54.584195 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584142    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-model-cache\") pod \"ddfe9255-2b02-43c1-b532-20641d7e3848\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") "
Jul 20 10:39:54.584195 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584179    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs\") pod \"ddfe9255-2b02-43c1-b532-20641d7e3848\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") "
Jul 20 10:39:54.584467 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584311    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") pod \"ddfe9255-2b02-43c1-b532-20641d7e3848\" (UID: \"ddfe9255-2b02-43c1-b532-20641d7e3848\") "
Jul 20 10:39:54.584467 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584421    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddfe9255-2b02-43c1-b532-20641d7e3848" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:54.584601 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584582    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:54.584827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.584786    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-home" (OuterVolumeSpecName: "home") pod "ddfe9255-2b02-43c1-b532-20641d7e3848" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:54.586528 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.586472    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfe9255-2b02-43c1-b532-20641d7e3848-kube-api-access-9r8f6" (OuterVolumeSpecName: "kube-api-access-9r8f6") pod "ddfe9255-2b02-43c1-b532-20641d7e3848" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848"). InnerVolumeSpecName "kube-api-access-9r8f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:39:54.586734 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.586713    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-dshm" (OuterVolumeSpecName: "dshm") pod "ddfe9255-2b02-43c1-b532-20641d7e3848" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:54.586882 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.586861    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddfe9255-2b02-43c1-b532-20641d7e3848" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:39:54.587217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.587200    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571" (OuterVolumeSpecName: "kserve-pvc-source") pod "ddfe9255-2b02-43c1-b532-20641d7e3848" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848"). InnerVolumeSpecName "pvc-c758418e-53f5-477b-bdd9-aecc24c01911". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 20 10:39:54.600723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.600690    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddfe9255-2b02-43c1-b532-20641d7e3848" (UID: "ddfe9255-2b02-43c1-b532-20641d7e3848"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:54.685746 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.685710    2572 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") on node \"ip-10-0-138-196.ec2.internal\" "
Jul 20 10:39:54.685746 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.685739    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:54.685746 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.685751    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:54.685957 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.685760    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddfe9255-2b02-43c1-b532-20641d7e3848-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:54.685957 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.685768    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8f6\" (UniqueName: \"kubernetes.io/projected/ddfe9255-2b02-43c1-b532-20641d7e3848-kube-api-access-9r8f6\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:54.685957 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.685776    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddfe9255-2b02-43c1-b532-20641d7e3848-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:39:54.714168 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.714150    2572 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-c758418e-53f5-477b-bdd9-aecc24c01911" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571") on node "ip-10-0-138-196.ec2.internal"
Jul 20 10:39:54.786132 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.786099    2572 reconciler_common.go:299] "Volume detached for volume \"pvc-c758418e-53f5-477b-bdd9-aecc24c01911\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05f9ab1c29c16f571\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"csi-52142cb8d77ff7ede58a62cde42b286fed755369dd19ccc158de8d922a9c65e2\""
Jul 20 10:39:54.854000 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.853895    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 20 10:39:54.855611 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:54.855587    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 20 10:39:56.519963 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:39:56.519925    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddfe9255-2b02-43c1-b532-20641d7e3848" path="/var/lib/kubelet/pods/ddfe9255-2b02-43c1-b532-20641d7e3848/volumes"
Jul 20 10:40:02.573822 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:02.573785    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"]
Jul 20 10:40:02.580109 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:02.580085    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-m9fbp"]
Jul 20 10:40:04.519481 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:04.519441    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e03b09f0-2ba7-4dd7-9d48-0818ffa18033" path="/var/lib/kubelet/pods/e03b09f0-2ba7-4dd7-9d48-0818ffa18033/volumes"
Jul 20 10:40:07.918229 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.918188    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"]
Jul 20 10:40:07.918726 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.918627    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main"
Jul 20 10:40:07.918726 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.918647    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main"
Jul 20 10:40:07.918726 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.918699    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddfe9255-2b02-43c1-b532-20641d7e3848" containerName="main"
Jul 20 10:40:07.918726 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.918708    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddfe9255-2b02-43c1-b532-20641d7e3848" containerName="main"
Jul 20 10:40:07.918941 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.918776    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b021d8-977a-406c-bf43-a70d0e7ac03b" containerName="main"
Jul 20 10:40:07.918941 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.918789    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddfe9255-2b02-43c1-b532-20641d7e3848" containerName="main"
Jul 20 10:40:07.921861 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.921840    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:07.925467 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.925440    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Jul 20 10:40:07.925601 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.925499    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Jul 20 10:40:07.925686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.925615    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-mfhh7\""
Jul 20 10:40:07.927604 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.927574    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Jul 20 10:40:07.927939 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.927916    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Jul 20 10:40:07.938243 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:07.938219    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"]
Jul 20 10:40:08.087348 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.087308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e5ec10-6e34-4370-9c9d-75881f0152b5-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.087348 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.087351    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-model-cache\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.087581 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.087367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-home\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.087581 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.087456    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.087581 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.087497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-dshm\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.087581 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.087541    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.087759 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.087619    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4qqg\" (UniqueName: \"kubernetes.io/projected/33e5ec10-6e34-4370-9c9d-75881f0152b5-kube-api-access-p4qqg\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188190 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-dshm\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188190 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188139    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188190 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4qqg\" (UniqueName: \"kubernetes.io/projected/33e5ec10-6e34-4370-9c9d-75881f0152b5-kube-api-access-p4qqg\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188434 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188203    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e5ec10-6e34-4370-9c9d-75881f0152b5-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188434 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188221    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-model-cache\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188434 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-home\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188434 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188255    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188661 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188641    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188725 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188710    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-home\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188783 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188742    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-model-cache\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.188836 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.188802    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.190501 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.190477    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-dshm\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.190744 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.190726    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e5ec10-6e34-4370-9c9d-75881f0152b5-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.196729 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.196701    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4qqg\" (UniqueName: \"kubernetes.io/projected/33e5ec10-6e34-4370-9c9d-75881f0152b5-kube-api-access-p4qqg\") pod \"e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.237653 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.237621    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:08.331233 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.331189    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"]
Jul 20 10:40:08.334723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.334702    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.337698 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.337636    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-95x2s\""
Jul 20 10:40:08.346076 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.346049    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"]
Jul 20 10:40:08.382572 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.382458    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"]
Jul 20 10:40:08.385427 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:40:08.385403    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e5ec10_6e34_4370_9c9d_75881f0152b5.slice/crio-89781cc733dd84879b352572f024e4fcf98bdc64275c2826dd5f8be8eac97be9 WatchSource:0}: Error finding container 89781cc733dd84879b352572f024e4fcf98bdc64275c2826dd5f8be8eac97be9: Status 404 returned error can't find the container with id 89781cc733dd84879b352572f024e4fcf98bdc64275c2826dd5f8be8eac97be9
Jul 20 10:40:08.491173 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.491139    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.491355 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.491205    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tokenizer-uds\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.491355 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.491251    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.491355 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.491316    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552z5\" (UniqueName: \"kubernetes.io/projected/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kube-api-access-552z5\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.586794 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.586760    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h" event={"ID":"33e5ec10-6e34-4370-9c9d-75881f0152b5","Type":"ContainerStarted","Data":"80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583"}
Jul 20 10:40:08.586958 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.586810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h" event={"ID":"33e5ec10-6e34-4370-9c9d-75881f0152b5","Type":"ContainerStarted","Data":"89781cc733dd84879b352572f024e4fcf98bdc64275c2826dd5f8be8eac97be9"}
Jul 20 10:40:08.592551 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.592530    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.592632 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.592585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tokenizer-uds\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.592632 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.592610    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.592727 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.592648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552z5\" (UniqueName: \"kubernetes.io/projected/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kube-api-access-552z5\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.592985 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.592968    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.593068 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.593021    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tokenizer-uds\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.595088 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.595072    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.600413 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.600392    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-552z5\" (UniqueName: \"kubernetes.io/projected/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kube-api-access-552z5\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.648901 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.648862    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:08.785022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:08.784942    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"]
Jul 20 10:40:08.787862 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:40:08.787832    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8870614_1afb_4ba3_9144_fc8b64c7d5d2.slice/crio-8e5940d69ed59defdfc7877318fdd2eb06425a95a5d6b7a52e4f7e98ecdf526a WatchSource:0}: Error finding container 8e5940d69ed59defdfc7877318fdd2eb06425a95a5d6b7a52e4f7e98ecdf526a: Status 404 returned error can't find the container with id 8e5940d69ed59defdfc7877318fdd2eb06425a95a5d6b7a52e4f7e98ecdf526a
Jul 20 10:40:09.595197 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:09.595167    2572 generic.go:356] "Generic (PLEG): container finished" podID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" containerID="6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7" exitCode=0
Jul 20 10:40:09.595591 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:09.595272    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" event={"ID":"f8870614-1afb-4ba3-9144-fc8b64c7d5d2","Type":"ContainerDied","Data":"6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7"}
Jul 20 10:40:09.595591 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:09.595335    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" event={"ID":"f8870614-1afb-4ba3-9144-fc8b64c7d5d2","Type":"ContainerStarted","Data":"8e5940d69ed59defdfc7877318fdd2eb06425a95a5d6b7a52e4f7e98ecdf526a"}
Jul 20 10:40:10.604300 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:10.604266    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" event={"ID":"f8870614-1afb-4ba3-9144-fc8b64c7d5d2","Type":"ContainerStarted","Data":"c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592"}
Jul 20 10:40:10.604850 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:10.604355    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:10.636779 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:10.636716    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" podStartSLOduration=1.911267272 podStartE2EDuration="2.636695208s" totalImagesPullingTime="0s" totalInitContainerRuntime="725.427936ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:40:10.634775529 +0000 UTC m=+2590.647877423" watchObservedRunningTime="2026-07-20 10:40:10.636695208 +0000 UTC m=+2590.649797104"
Jul 20 10:40:11.153625 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.153591    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"]
Jul 20 10:40:11.159020 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.158991    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"]
Jul 20 10:40:11.159227 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.159203    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h" podUID="33e5ec10-6e34-4370-9c9d-75881f0152b5" containerName="storage-initializer" containerID="cri-o://80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583" gracePeriod=60
Jul 20 10:40:11.608433 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.608398    2572 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" secret="" err="secret \"e2e-del-blocked-svc-epp-sa-dockercfg-95x2s\" not found"
Jul 20 10:40:11.719176 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:11.719147    2572 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:11.719335 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:11.719215    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs podName:f8870614-1afb-4ba3-9144-fc8b64c7d5d2 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:12.219200168 +0000 UTC m=+2592.232302041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:11.815641 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.815615    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:11.920723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.920635    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-kserve-provision-location\") pod \"33e5ec10-6e34-4370-9c9d-75881f0152b5\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") "
Jul 20 10:40:11.920723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.920701    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-tmp-dir\") pod \"33e5ec10-6e34-4370-9c9d-75881f0152b5\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") "
Jul 20 10:40:11.920940 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.920731    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-home\") pod \"33e5ec10-6e34-4370-9c9d-75881f0152b5\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") "
Jul 20 10:40:11.920940 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.920765    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-model-cache\") pod \"33e5ec10-6e34-4370-9c9d-75881f0152b5\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") "
Jul 20 10:40:11.920940 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.920809    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-dshm\") pod \"33e5ec10-6e34-4370-9c9d-75881f0152b5\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") "
Jul 20 10:40:11.920940 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.920852    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e5ec10-6e34-4370-9c9d-75881f0152b5-tls-certs\") pod \"33e5ec10-6e34-4370-9c9d-75881f0152b5\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") "
Jul 20 10:40:11.920940 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.920874    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4qqg\" (UniqueName: \"kubernetes.io/projected/33e5ec10-6e34-4370-9c9d-75881f0152b5-kube-api-access-p4qqg\") pod \"33e5ec10-6e34-4370-9c9d-75881f0152b5\" (UID: \"33e5ec10-6e34-4370-9c9d-75881f0152b5\") "
Jul 20 10:40:11.921213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.921012    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33e5ec10-6e34-4370-9c9d-75881f0152b5" (UID: "33e5ec10-6e34-4370-9c9d-75881f0152b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:11.921213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.921023    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-home" (OuterVolumeSpecName: "home") pod "33e5ec10-6e34-4370-9c9d-75881f0152b5" (UID: "33e5ec10-6e34-4370-9c9d-75881f0152b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:11.921213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.921171    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "33e5ec10-6e34-4370-9c9d-75881f0152b5" (UID: "33e5ec10-6e34-4370-9c9d-75881f0152b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:11.921213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.921186    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:11.921213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.921205    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:11.923064 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.923036    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-dshm" (OuterVolumeSpecName: "dshm") pod "33e5ec10-6e34-4370-9c9d-75881f0152b5" (UID: "33e5ec10-6e34-4370-9c9d-75881f0152b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:11.923176 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.923094    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e5ec10-6e34-4370-9c9d-75881f0152b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33e5ec10-6e34-4370-9c9d-75881f0152b5" (UID: "33e5ec10-6e34-4370-9c9d-75881f0152b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:40:11.923222 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.923208    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e5ec10-6e34-4370-9c9d-75881f0152b5-kube-api-access-p4qqg" (OuterVolumeSpecName: "kube-api-access-p4qqg") pod "33e5ec10-6e34-4370-9c9d-75881f0152b5" (UID: "33e5ec10-6e34-4370-9c9d-75881f0152b5"). InnerVolumeSpecName "kube-api-access-p4qqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:40:11.994649 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:11.994608    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33e5ec10-6e34-4370-9c9d-75881f0152b5" (UID: "33e5ec10-6e34-4370-9c9d-75881f0152b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:12.022016 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.021980    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:12.022016 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.022010    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:12.022016 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.022019    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e5ec10-6e34-4370-9c9d-75881f0152b5-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:12.022248 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.022027    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4qqg\" (UniqueName: \"kubernetes.io/projected/33e5ec10-6e34-4370-9c9d-75881f0152b5-kube-api-access-p4qqg\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:12.022248 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.022035    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e5ec10-6e34-4370-9c9d-75881f0152b5-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:12.224791 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:12.224757    2572 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:12.224960 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:12.224833    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs podName:f8870614-1afb-4ba3-9144-fc8b64c7d5d2 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:13.224818248 +0000 UTC m=+2593.237920121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:12.614977 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.614939    2572 generic.go:356] "Generic (PLEG): container finished" podID="33e5ec10-6e34-4370-9c9d-75881f0152b5" containerID="80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583" exitCode=0
Jul 20 10:40:12.615379 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.614991    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"
Jul 20 10:40:12.615379 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.614989    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h" event={"ID":"33e5ec10-6e34-4370-9c9d-75881f0152b5","Type":"ContainerDied","Data":"80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583"}
Jul 20 10:40:12.615379 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.615048    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h" event={"ID":"33e5ec10-6e34-4370-9c9d-75881f0152b5","Type":"ContainerDied","Data":"89781cc733dd84879b352572f024e4fcf98bdc64275c2826dd5f8be8eac97be9"}
Jul 20 10:40:12.615379 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.615069    2572 scope.go:117] "RemoveContainer" containerID="80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583"
Jul 20 10:40:12.623397 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.623379    2572 scope.go:117] "RemoveContainer" containerID="80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583"
Jul 20 10:40:12.623687 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:12.623668    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583\": container with ID starting with 80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583 not found: ID does not exist" containerID="80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583"
Jul 20 10:40:12.623762 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.623699    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583"} err="failed to get container status \"80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583\": rpc error: code = NotFound desc = could not find container \"80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583\": container with ID starting with 80ca54f7d1d9e4350885bbdd4796316c758dea05c940c70a65ef8abeaf951583 not found: ID does not exist"
Jul 20 10:40:12.714694 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.714666    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"]
Jul 20 10:40:12.721203 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:12.721172    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fd4989cb8-w4h6h"]
Jul 20 10:40:13.233623 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:13.233592    2572 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:13.233815 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:13.233669    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs podName:f8870614-1afb-4ba3-9144-fc8b64c7d5d2 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:15.233653986 +0000 UTC m=+2595.246755860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:14.519129 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:14.519088    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e5ec10-6e34-4370-9c9d-75881f0152b5" path="/var/lib/kubelet/pods/33e5ec10-6e34-4370-9c9d-75881f0152b5/volumes"
Jul 20 10:40:15.246857 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:15.246829    2572 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:15.247021 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:15.246890    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs podName:f8870614-1afb-4ba3-9144-fc8b64c7d5d2 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:19.246875056 +0000 UTC m=+2599.259976930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:19.278775 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:19.278728    2572 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:19.279214 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:19.278833    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs podName:f8870614-1afb-4ba3-9144-fc8b64c7d5d2 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:27.278819077 +0000 UTC m=+2607.291920951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:27.345463 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:27.345423    2572 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:27.345921 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:27.345497    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs podName:f8870614-1afb-4ba3-9144-fc8b64c7d5d2 nodeName:}" failed. No retries permitted until 2026-07-20 10:40:43.345480831 +0000 UTC m=+2623.358582705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Jul 20 10:40:27.636598 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.636484    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" podUID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" containerName="main" containerID="cri-o://c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592" gracePeriod=45
Jul 20 10:40:27.879617 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.879592    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:27.949004 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.948982    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tokenizer-uds\") pod \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") "
Jul 20 10:40:27.949194 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.949040    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-552z5\" (UniqueName: \"kubernetes.io/projected/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kube-api-access-552z5\") pod \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") "
Jul 20 10:40:27.949194 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.949063    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs\") pod \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") "
Jul 20 10:40:27.949194 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.949086    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kserve-provision-location\") pod \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\" (UID: \"f8870614-1afb-4ba3-9144-fc8b64c7d5d2\") "
Jul 20 10:40:27.949366 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.949285    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f8870614-1afb-4ba3-9144-fc8b64c7d5d2" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:27.949789 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.949766    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8870614-1afb-4ba3-9144-fc8b64c7d5d2" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:27.951216 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.951195    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kube-api-access-552z5" (OuterVolumeSpecName: "kube-api-access-552z5") pod "f8870614-1afb-4ba3-9144-fc8b64c7d5d2" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2"). InnerVolumeSpecName "kube-api-access-552z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:40:27.951306 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:27.951226    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8870614-1afb-4ba3-9144-fc8b64c7d5d2" (UID: "f8870614-1afb-4ba3-9144-fc8b64c7d5d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:40:28.050372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.050325    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:28.050372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.050363    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552z5\" (UniqueName: \"kubernetes.io/projected/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kube-api-access-552z5\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:28.050372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.050372    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:28.050372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.050381    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8870614-1afb-4ba3-9144-fc8b64c7d5d2-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:28.178270 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178235    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"]
Jul 20 10:40:28.178538 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178527    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" containerName="main"
Jul 20 10:40:28.178619 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178539    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" containerName="main"
Jul 20 10:40:28.178619 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178549    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" containerName="storage-initializer"
Jul 20 10:40:28.178619 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178569    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" containerName="storage-initializer"
Jul 20 10:40:28.178619 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178585    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e5ec10-6e34-4370-9c9d-75881f0152b5" containerName="storage-initializer"
Jul 20 10:40:28.178619 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178591    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e5ec10-6e34-4370-9c9d-75881f0152b5" containerName="storage-initializer"
Jul 20 10:40:28.178816 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178644    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e5ec10-6e34-4370-9c9d-75881f0152b5" containerName="storage-initializer"
Jul 20 10:40:28.178816 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.178651    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" containerName="main"
Jul 20 10:40:28.181830 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.181808    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:28.184571 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.184536    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Jul 20 10:40:28.184690 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.184544    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Jul 20 10:40:28.184690 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.184609    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-qs5vk\""
Jul 20 10:40:28.184844 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.184829    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Jul 20 10:40:28.191307 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.191288    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"]
Jul 20 10:40:28.251718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.251625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb\" (UID: \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:28.251718 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.251704    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwgh\" (UniqueName: \"kubernetes.io/projected/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kube-api-access-gzwgh\") pod \"e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb\" (UID: \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:28.352955 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.352923    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb\" (UID: \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:28.353408 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.352983    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgh\" (UniqueName: \"kubernetes.io/projected/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kube-api-access-gzwgh\") pod \"e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb\" (UID: \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:28.353408 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.353322    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb\" (UID: \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:28.361892 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.361865    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwgh\" (UniqueName: \"kubernetes.io/projected/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kube-api-access-gzwgh\") pod \"e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb\" (UID: \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:28.493037 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.492998    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:28.617953 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.617924    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"]
Jul 20 10:40:28.620937 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:40:28.620912    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6998c962_8410_40f7_a9d0_2ca240e8f1b8.slice/crio-d9885ee94212c0f18a4714c44d1e2bc61f78f6e1b4eadf57ffe8f2dd7fe13c7a WatchSource:0}: Error finding container d9885ee94212c0f18a4714c44d1e2bc61f78f6e1b4eadf57ffe8f2dd7fe13c7a: Status 404 returned error can't find the container with id d9885ee94212c0f18a4714c44d1e2bc61f78f6e1b4eadf57ffe8f2dd7fe13c7a
Jul 20 10:40:28.686414 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.686383    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb" event={"ID":"6998c962-8410-40f7-a9d0-2ca240e8f1b8","Type":"ContainerStarted","Data":"d9885ee94212c0f18a4714c44d1e2bc61f78f6e1b4eadf57ffe8f2dd7fe13c7a"}
Jul 20 10:40:28.687974 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.687949    2572 generic.go:356] "Generic (PLEG): container finished" podID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" containerID="c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592" exitCode=0
Jul 20 10:40:28.688084 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.688010    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"
Jul 20 10:40:28.688084 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.688025    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" event={"ID":"f8870614-1afb-4ba3-9144-fc8b64c7d5d2","Type":"ContainerDied","Data":"c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592"}
Jul 20 10:40:28.688084 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.688061    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7" event={"ID":"f8870614-1afb-4ba3-9144-fc8b64c7d5d2","Type":"ContainerDied","Data":"8e5940d69ed59defdfc7877318fdd2eb06425a95a5d6b7a52e4f7e98ecdf526a"}
Jul 20 10:40:28.688220 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.688087    2572 scope.go:117] "RemoveContainer" containerID="c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592"
Jul 20 10:40:28.698426 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.698404    2572 scope.go:117] "RemoveContainer" containerID="6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7"
Jul 20 10:40:28.706223 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.706188    2572 scope.go:117] "RemoveContainer" containerID="c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592"
Jul 20 10:40:28.706487 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:28.706464    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592\": container with ID starting with c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592 not found: ID does not exist" containerID="c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592"
Jul 20 10:40:28.706624 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.706491    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592"} err="failed to get container status \"c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592\": rpc error: code = NotFound desc = could not find container \"c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592\": container with ID starting with c30bdfd2db22980933c076fe38dbdb16db9d793d0349cee2adfd26e035f0f592 not found: ID does not exist"
Jul 20 10:40:28.706624 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.706509    2572 scope.go:117] "RemoveContainer" containerID="6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7"
Jul 20 10:40:28.706787 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:28.706766    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7\": container with ID starting with 6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7 not found: ID does not exist" containerID="6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7"
Jul 20 10:40:28.706853 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.706790    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7"} err="failed to get container status \"6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7\": rpc error: code = NotFound desc = could not find container \"6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7\": container with ID starting with 6b39f68a1837c60bd83c281542b226760390602c3842a60b090ebabe995ccfa7 not found: ID does not exist"
Jul 20 10:40:28.710873 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.710852    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"]
Jul 20 10:40:28.716296 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:28.716274    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86786fd98f-bx6v7"]
Jul 20 10:40:29.695546 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:29.695499    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb" event={"ID":"6998c962-8410-40f7-a9d0-2ca240e8f1b8","Type":"ContainerStarted","Data":"1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4"}
Jul 20 10:40:30.522850 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:30.522755    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8870614-1afb-4ba3-9144-fc8b64c7d5d2" path="/var/lib/kubelet/pods/f8870614-1afb-4ba3-9144-fc8b64c7d5d2/volumes"
Jul 20 10:40:31.286437 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:31.286405    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"]
Jul 20 10:40:31.712569 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:31.712501    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb" podUID="6998c962-8410-40f7-a9d0-2ca240e8f1b8" containerName="storage-initializer" containerID="cri-o://1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4" gracePeriod=30
Jul 20 10:40:32.462715 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.462690    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:32.485906 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.485870    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kserve-provision-location\") pod \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\" (UID: \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\") "
Jul 20 10:40:32.486063 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.485925    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzwgh\" (UniqueName: \"kubernetes.io/projected/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kube-api-access-gzwgh\") pod \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\" (UID: \"6998c962-8410-40f7-a9d0-2ca240e8f1b8\") "
Jul 20 10:40:32.488191 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.488162    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kube-api-access-gzwgh" (OuterVolumeSpecName: "kube-api-access-gzwgh") pod "6998c962-8410-40f7-a9d0-2ca240e8f1b8" (UID: "6998c962-8410-40f7-a9d0-2ca240e8f1b8"). InnerVolumeSpecName "kube-api-access-gzwgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:40:32.562477 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.562441    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6998c962-8410-40f7-a9d0-2ca240e8f1b8" (UID: "6998c962-8410-40f7-a9d0-2ca240e8f1b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:32.586624 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.586585    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:32.586624 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.586618    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzwgh\" (UniqueName: \"kubernetes.io/projected/6998c962-8410-40f7-a9d0-2ca240e8f1b8-kube-api-access-gzwgh\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:40:32.718169 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.718130    2572 generic.go:356] "Generic (PLEG): container finished" podID="6998c962-8410-40f7-a9d0-2ca240e8f1b8" containerID="1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4" exitCode=0
Jul 20 10:40:32.718375 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.718169    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb" event={"ID":"6998c962-8410-40f7-a9d0-2ca240e8f1b8","Type":"ContainerDied","Data":"1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4"}
Jul 20 10:40:32.718375 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.718227    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb" event={"ID":"6998c962-8410-40f7-a9d0-2ca240e8f1b8","Type":"ContainerDied","Data":"d9885ee94212c0f18a4714c44d1e2bc61f78f6e1b4eadf57ffe8f2dd7fe13c7a"}
Jul 20 10:40:32.718375 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.718178    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"
Jul 20 10:40:32.718375 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.718254    2572 scope.go:117] "RemoveContainer" containerID="1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4"
Jul 20 10:40:32.726756 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.726738    2572 scope.go:117] "RemoveContainer" containerID="1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4"
Jul 20 10:40:32.727006 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:40:32.726989    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4\": container with ID starting with 1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4 not found: ID does not exist" containerID="1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4"
Jul 20 10:40:32.727049 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.727014    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4"} err="failed to get container status \"1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4\": rpc error: code = NotFound desc = could not find container \"1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4\": container with ID starting with 1608af2dcc44e8eadef46c40a38fc183fcea7940047622b1dcb381d70357f2b4 not found: ID does not exist"
Jul 20 10:40:32.748945 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.748876    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"]
Jul 20 10:40:32.752751 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:32.752727    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-5f46447dc7-b27zb"]
Jul 20 10:40:34.519530 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:34.519495    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6998c962-8410-40f7-a9d0-2ca240e8f1b8" path="/var/lib/kubelet/pods/6998c962-8410-40f7-a9d0-2ca240e8f1b8/volumes"
Jul 20 10:40:47.713403 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.713368    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"]
Jul 20 10:40:47.713860 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.713725    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6998c962-8410-40f7-a9d0-2ca240e8f1b8" containerName="storage-initializer"
Jul 20 10:40:47.713860 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.713738    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6998c962-8410-40f7-a9d0-2ca240e8f1b8" containerName="storage-initializer"
Jul 20 10:40:47.713860 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.713789    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6998c962-8410-40f7-a9d0-2ca240e8f1b8" containerName="storage-initializer"
Jul 20 10:40:47.717084 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.717062    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.720013 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.719989    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 20 10:40:47.720247 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.720194    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 20 10:40:47.720247 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.720233    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 20 10:40:47.720426 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.720261    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 20 10:40:47.720426 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.720383    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-q567k\""
Jul 20 10:40:47.728707 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.728683    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"]
Jul 20 10:40:47.813503 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.813471    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-dshm\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.813503 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.813505    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kserve-provision-location\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.813731 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.813524    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmks\" (UniqueName: \"kubernetes.io/projected/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kube-api-access-2nmks\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.813731 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.813583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tmp-dir\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.813731 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.813661    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-home\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.813838 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.813733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-model-cache\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.813838 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.813752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tls-certs\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.914687 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.914658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-model-cache\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.914687 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.914689    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tls-certs\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.914876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.914718    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-dshm\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.914876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.914847    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kserve-provision-location\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.914967 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.914880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmks\" (UniqueName: \"kubernetes.io/projected/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kube-api-access-2nmks\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.914967 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.914927    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tmp-dir\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.915152 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.914987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-home\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.915212 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.915155    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-model-cache\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.915264 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.915225    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kserve-provision-location\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.915307 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.915294    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tmp-dir\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.915355 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.915343    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-home\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.917069 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.917049    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-dshm\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.917255 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.917239    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tls-certs\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.929188 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.929158    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmks\" (UniqueName: \"kubernetes.io/projected/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kube-api-access-2nmks\") pod \"stop-feature-test-kserve-cd9f9b479-bdjzd\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:47.930366 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.930342    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"]
Jul 20 10:40:47.934054 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.934034    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:47.937297 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.937275    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-mkqq4\""
Jul 20 10:40:47.945697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:47.945673    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"]
Jul 20 10:40:48.016208 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.016113    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.016208 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.016154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ns6\" (UniqueName: \"kubernetes.io/projected/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kube-api-access-d7ns6\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.016208 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.016186    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.016425 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.016230    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.032069 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.032037    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:48.117044 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.117013    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.117225 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.117066    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.117225 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.117110    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ns6\" (UniqueName: \"kubernetes.io/projected/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kube-api-access-d7ns6\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.117225 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.117174    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.117572 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.117535    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.117678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.117583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.119592 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.119570    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.127999 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.127974    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ns6\" (UniqueName: \"kubernetes.io/projected/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kube-api-access-d7ns6\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.168407 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.168383    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"]
Jul 20 10:40:48.170197 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:40:48.170172    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10c2e59_7f74_4654_9ff6_dd5addaf38e1.slice/crio-80925774d066e4edc2fc0ad78e835012124f72271c0974a196ef9dd0867c96e4 WatchSource:0}: Error finding container 80925774d066e4edc2fc0ad78e835012124f72271c0974a196ef9dd0867c96e4: Status 404 returned error can't find the container with id 80925774d066e4edc2fc0ad78e835012124f72271c0974a196ef9dd0867c96e4
Jul 20 10:40:48.247785 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.247768    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:48.376791 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.376759    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"]
Jul 20 10:40:48.379881 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:40:48.379849    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5f22e5_86b5_4346_a1e7_cd2e8eceec3d.slice/crio-a768fd15959dbde00678eeea5f05dfc8b7bd7fca56e4fd9682360fd2352a54fa WatchSource:0}: Error finding container a768fd15959dbde00678eeea5f05dfc8b7bd7fca56e4fd9682360fd2352a54fa: Status 404 returned error can't find the container with id a768fd15959dbde00678eeea5f05dfc8b7bd7fca56e4fd9682360fd2352a54fa
Jul 20 10:40:48.786615 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.786578    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" event={"ID":"f10c2e59-7f74-4654-9ff6-dd5addaf38e1","Type":"ContainerStarted","Data":"65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6"}
Jul 20 10:40:48.787078 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.786637    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" event={"ID":"f10c2e59-7f74-4654-9ff6-dd5addaf38e1","Type":"ContainerStarted","Data":"80925774d066e4edc2fc0ad78e835012124f72271c0974a196ef9dd0867c96e4"}
Jul 20 10:40:48.788219 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.788190    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" event={"ID":"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d","Type":"ContainerStarted","Data":"4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f"}
Jul 20 10:40:48.788319 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:48.788245    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" event={"ID":"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d","Type":"ContainerStarted","Data":"a768fd15959dbde00678eeea5f05dfc8b7bd7fca56e4fd9682360fd2352a54fa"}
Jul 20 10:40:49.793855 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:49.793812    2572 generic.go:356] "Generic (PLEG): container finished" podID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerID="4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f" exitCode=0
Jul 20 10:40:49.794319 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:49.793895    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" event={"ID":"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d","Type":"ContainerDied","Data":"4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f"}
Jul 20 10:40:50.800384 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:50.800348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" event={"ID":"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d","Type":"ContainerStarted","Data":"db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece"}
Jul 20 10:40:50.800779 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:50.800462    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:40:50.822071 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:50.822014    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" podStartSLOduration=3.171825804 podStartE2EDuration="3.821999383s" totalImagesPullingTime="0s" totalInitContainerRuntime="650.173579ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:40:50.821820258 +0000 UTC m=+2630.834922167" watchObservedRunningTime="2026-07-20 10:40:50.821999383 +0000 UTC m=+2630.835101281"
Jul 20 10:40:51.806143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:51.806057    2572 generic.go:356] "Generic (PLEG): container finished" podID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerID="65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6" exitCode=0
Jul 20 10:40:51.806143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:51.806130    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" event={"ID":"f10c2e59-7f74-4654-9ff6-dd5addaf38e1","Type":"ContainerDied","Data":"65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6"}
Jul 20 10:40:52.812985 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:52.812952    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" event={"ID":"f10c2e59-7f74-4654-9ff6-dd5addaf38e1","Type":"ContainerStarted","Data":"f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93"}
Jul 20 10:40:52.838840 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:52.838781    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podStartSLOduration=2.595312147 podStartE2EDuration="5.838761126s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.243448979s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:40:52.835906083 +0000 UTC m=+2632.849008004" watchObservedRunningTime="2026-07-20 10:40:52.838761126 +0000 UTC m=+2632.851863023"
Jul 20 10:40:53.032335 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:53.032299    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:58.033189 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:58.033145    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:40:58.034958 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:40:58.034926    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:41:00.757868 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:41:00.757837    2572 scope.go:117] "RemoveContainer" containerID="aac56d1e6f3c3d8eed8619447220e531591e0b8e7795bc880606970442b9a53f"
Jul 20 10:41:08.033125 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:41:08.033074    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:41:18.033088 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:41:18.033040    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:41:21.808214 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:41:21.808179    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:41:28.032767 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:41:28.032724    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:41:38.032939 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:41:38.032885    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:41:48.032860 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:41:48.032817    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:41:58.032689 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:41:58.032582    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:42:00.586815 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:00.586784    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:42:00.599978 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:00.599952    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:42:08.033139 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:08.033081    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:42:18.032891 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:18.032843    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:42:28.033323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:28.033280    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:42:38.032837 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:38.032791    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:42:48.042693 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:48.042661    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:42:48.050394 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:48.050372    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:42:49.390015 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:49.389981    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"]
Jul 20 10:42:49.596433 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:49.596390    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"]
Jul 20 10:42:51.808469 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:42:51.808422    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:43:01.808298 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:01.808260    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:43:04.413149 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.413101    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" containerID="cri-o://f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93" gracePeriod=45
Jul 20 10:43:04.616447 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.616383    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerName="main" containerID="cri-o://db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece" gracePeriod=45
Jul 20 10:43:04.873288 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.873253    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:43:04.979550 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.979516    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kserve-provision-location\") pod \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") "
Jul 20 10:43:04.979727 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.979594    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tls-certs\") pod \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") "
Jul 20 10:43:04.979727 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.979627    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tokenizer-uds\") pod \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") "
Jul 20 10:43:04.979727 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.979656    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7ns6\" (UniqueName: \"kubernetes.io/projected/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kube-api-access-d7ns6\") pod \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\" (UID: \"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d\") "
Jul 20 10:43:04.979932 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.979908    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" (UID: "8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:43:04.980313 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.980288    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" (UID: "8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:43:04.981910 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.981884    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" (UID: "8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:43:04.981994 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:04.981925    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kube-api-access-d7ns6" (OuterVolumeSpecName: "kube-api-access-d7ns6") pod "8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" (UID: "8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d"). InnerVolumeSpecName "kube-api-access-d7ns6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:43:05.043608 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.043568    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 20 10:43:05.081141 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.081060    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.081141 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.081083    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.081141 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.081092    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7ns6\" (UniqueName: \"kubernetes.io/projected/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kube-api-access-d7ns6\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.081141 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.081102    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.154021 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.153996    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:43:05.182202 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.182171    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-home\") pod \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") "
Jul 20 10:43:05.182380 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.182231    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nmks\" (UniqueName: \"kubernetes.io/projected/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kube-api-access-2nmks\") pod \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") "
Jul 20 10:43:05.182380 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.182270    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tmp-dir\") pod \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") "
Jul 20 10:43:05.182380 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.182298    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-model-cache\") pod \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") "
Jul 20 10:43:05.182380 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.182356    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-dshm\") pod \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") "
Jul 20 10:43:05.182618 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.182380    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tls-certs\") pod \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") "
Jul 20 10:43:05.182618 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.182404    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kserve-provision-location\") pod \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\" (UID: \"f10c2e59-7f74-4654-9ff6-dd5addaf38e1\") "
Jul 20 10:43:05.183098 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.183054    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-home" (OuterVolumeSpecName: "home") pod "f10c2e59-7f74-4654-9ff6-dd5addaf38e1" (UID: "f10c2e59-7f74-4654-9ff6-dd5addaf38e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:43:05.183346 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.183315    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f10c2e59-7f74-4654-9ff6-dd5addaf38e1" (UID: "f10c2e59-7f74-4654-9ff6-dd5addaf38e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:43:05.185089 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.185059    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f10c2e59-7f74-4654-9ff6-dd5addaf38e1" (UID: "f10c2e59-7f74-4654-9ff6-dd5addaf38e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:43:05.185213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.185071    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kube-api-access-2nmks" (OuterVolumeSpecName: "kube-api-access-2nmks") pod "f10c2e59-7f74-4654-9ff6-dd5addaf38e1" (UID: "f10c2e59-7f74-4654-9ff6-dd5addaf38e1"). InnerVolumeSpecName "kube-api-access-2nmks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:43:05.185323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.185288    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-dshm" (OuterVolumeSpecName: "dshm") pod "f10c2e59-7f74-4654-9ff6-dd5addaf38e1" (UID: "f10c2e59-7f74-4654-9ff6-dd5addaf38e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:43:05.194621 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.194592    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f10c2e59-7f74-4654-9ff6-dd5addaf38e1" (UID: "f10c2e59-7f74-4654-9ff6-dd5addaf38e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:43:05.256052 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.255957    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f10c2e59-7f74-4654-9ff6-dd5addaf38e1" (UID: "f10c2e59-7f74-4654-9ff6-dd5addaf38e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:43:05.283418 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.283384    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.283418 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.283413    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nmks\" (UniqueName: \"kubernetes.io/projected/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kube-api-access-2nmks\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.283418 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.283426    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.283686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.283439    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.283686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.283449    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.283686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.283459    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.283686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.283469    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10c2e59-7f74-4654-9ff6-dd5addaf38e1-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:43:05.367490 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.367457    2572 generic.go:356] "Generic (PLEG): container finished" podID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerID="db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece" exitCode=0
Jul 20 10:43:05.367685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.367510    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"
Jul 20 10:43:05.367685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.367528    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" event={"ID":"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d","Type":"ContainerDied","Data":"db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece"}
Jul 20 10:43:05.367685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.367610    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx" event={"ID":"8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d","Type":"ContainerDied","Data":"a768fd15959dbde00678eeea5f05dfc8b7bd7fca56e4fd9682360fd2352a54fa"}
Jul 20 10:43:05.367685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.367643    2572 scope.go:117] "RemoveContainer" containerID="db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece"
Jul 20 10:43:05.369394 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.369369    2572 generic.go:356] "Generic (PLEG): container finished" podID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerID="f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93" exitCode=0
Jul 20 10:43:05.369520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.369406    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"
Jul 20 10:43:05.369520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.369437    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" event={"ID":"f10c2e59-7f74-4654-9ff6-dd5addaf38e1","Type":"ContainerDied","Data":"f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93"}
Jul 20 10:43:05.369520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.369476    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd" event={"ID":"f10c2e59-7f74-4654-9ff6-dd5addaf38e1","Type":"ContainerDied","Data":"80925774d066e4edc2fc0ad78e835012124f72271c0974a196ef9dd0867c96e4"}
Jul 20 10:43:05.377834 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.377799    2572 scope.go:117] "RemoveContainer" containerID="4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f"
Jul 20 10:43:05.385817 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.385801    2572 scope.go:117] "RemoveContainer" containerID="db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece"
Jul 20 10:43:05.386077 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:43:05.386059    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece\": container with ID starting with db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece not found: ID does not exist" containerID="db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece"
Jul 20 10:43:05.386128 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.386087    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece"} err="failed to get container status \"db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece\": rpc error: code = NotFound desc = could not find container \"db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece\": container with ID starting with db4c7d49d43b99b7f03236dd4c3863c1620e99a02e8f10406a6b433ecfa28ece not found: ID does not exist"
Jul 20 10:43:05.386128 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.386103    2572 scope.go:117] "RemoveContainer" containerID="4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f"
Jul 20 10:43:05.386352 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:43:05.386334    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f\": container with ID starting with 4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f not found: ID does not exist" containerID="4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f"
Jul 20 10:43:05.386416 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.386362    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f"} err="failed to get container status \"4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f\": rpc error: code = NotFound desc = could not find container \"4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f\": container with ID starting with 4f522ed7f20ab76f1bd7656d7002bfd54c6c308afebe392f8cc51468f2fb5f4f not found: ID does not exist"
Jul 20 10:43:05.386416 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.386384    2572 scope.go:117] "RemoveContainer" containerID="f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93"
Jul 20 10:43:05.396166 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.396141    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"]
Jul 20 10:43:05.396427 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.396410    2572 scope.go:117] "RemoveContainer" containerID="65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6"
Jul 20 10:43:05.397746 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.397727    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-l9kwx"]
Jul 20 10:43:05.404361 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.404338    2572 scope.go:117] "RemoveContainer" containerID="f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93"
Jul 20 10:43:05.404628 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:43:05.404604    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93\": container with ID starting with f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93 not found: ID does not exist" containerID="f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93"
Jul 20 10:43:05.404715 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.404637    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93"} err="failed to get container status \"f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93\": rpc error: code = NotFound desc = could not find container \"f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93\": container with ID starting with f4c6a2568466bfd926bf27fda27d5fdc732604d7fa5c5c2814f8a179e660ac93 not found: ID does not exist"
Jul 20 10:43:05.404715 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.404658    2572 scope.go:117] "RemoveContainer" containerID="65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6"
Jul 20 10:43:05.404909 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:43:05.404891    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6\": container with ID starting with 65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6 not found: ID does not exist" containerID="65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6"
Jul 20 10:43:05.404966 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.404918    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6"} err="failed to get container status \"65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6\": rpc error: code = NotFound desc = could not find container \"65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6\": container with ID starting with 65cbb5056fb09ad10c46cf04cd55051a7cbf44d6bba6b5ef6903602ffb7509d6 not found: ID does not exist"
Jul 20 10:43:05.412505 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.412483    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"]
Jul 20 10:43:05.418764 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:05.418744    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-bdjzd"]
Jul 20 10:43:06.519156 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:06.519123    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" path="/var/lib/kubelet/pods/8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d/volumes"
Jul 20 10:43:06.519541 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:06.519527    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" path="/var/lib/kubelet/pods/f10c2e59-7f74-4654-9ff6-dd5addaf38e1/volumes"
Jul 20 10:43:21.870805 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.870772    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"]
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871046    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerName="main"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871056    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerName="main"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871072    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871077    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871085    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="storage-initializer"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871091    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="storage-initializer"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871102    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerName="storage-initializer"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871108    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerName="storage-initializer"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871160    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f10c2e59-7f74-4654-9ff6-dd5addaf38e1" containerName="main"
Jul 20 10:43:21.871206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.871168    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c5f22e5-86b5-4346-a1e7-cd2e8eceec3d" containerName="main"
Jul 20 10:43:21.874309 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.874289    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:21.879984 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.879924    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 20 10:43:21.880128 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.880028    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 20 10:43:21.880232 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.880195    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-q567k\""
Jul 20 10:43:21.880291 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.880220    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 20 10:43:21.880291 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.880280    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 20 10:43:21.897385 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.897323    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"]
Jul 20 10:43:21.901071 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.901044    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tls-certs\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:21.901235 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.901097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-dshm\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:21.901235 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.901164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqt7\" (UniqueName: \"kubernetes.io/projected/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kube-api-access-mcqt7\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:21.901235 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.901225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-model-cache\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:21.901430 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.901257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-home\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:21.901430 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.901303    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kserve-provision-location\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:21.901430 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:21.901349    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tmp-dir\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.001868 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.001824    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tls-certs\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.001868 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.001869    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-dshm\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002131 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.001901    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqt7\" (UniqueName: \"kubernetes.io/projected/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kube-api-access-mcqt7\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002131 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.001918    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-model-cache\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002131 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.001936    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-home\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002131 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.001955    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kserve-provision-location\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002131 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.001977    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tmp-dir\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002477 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.002452    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kserve-provision-location\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.002482    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-model-cache\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.002501    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-home\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.002711 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.002609    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tmp-dir\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.004277 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.004260    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-dshm\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.004547 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.004532    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tls-certs\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.019159 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.019132    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqt7\" (UniqueName: \"kubernetes.io/projected/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kube-api-access-mcqt7\") pod \"stop-feature-test-kserve-cd9f9b479-z549t\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.172919 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.172810    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"]
Jul 20 10:43:22.176191 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.176174    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.182611 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.182550    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-j5zcg\""
Jul 20 10:43:22.184278 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.184254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:22.199425 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.199395    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"]
Jul 20 10:43:22.204285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.204253    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpjb\" (UniqueName: \"kubernetes.io/projected/d160edff-b38f-42cc-9caf-6a75b62b7470-kube-api-access-jvpjb\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.204421 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.204368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d160edff-b38f-42cc-9caf-6a75b62b7470-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.204421 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.204399    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.204575 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.204489    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.305281 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.305246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d160edff-b38f-42cc-9caf-6a75b62b7470-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.305281 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.305284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.305537 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.305306    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.305537 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.305358    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpjb\" (UniqueName: \"kubernetes.io/projected/d160edff-b38f-42cc-9caf-6a75b62b7470-kube-api-access-jvpjb\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.305792 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.305766    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.305916 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.305819    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.308469 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.308447    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d160edff-b38f-42cc-9caf-6a75b62b7470-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.325094 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.325065    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpjb\" (UniqueName: \"kubernetes.io/projected/d160edff-b38f-42cc-9caf-6a75b62b7470-kube-api-access-jvpjb\") pod \"stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.352184 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.352149    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"]
Jul 20 10:43:22.355241 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:43:22.355212    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2237c8_af79_49c7_ad6a_0c5bea660d8d.slice/crio-3d6fd9f62cc6e29b9f554d1e18447a05d37dad79ce809ba9ee0603afcac26229 WatchSource:0}: Error finding container 3d6fd9f62cc6e29b9f554d1e18447a05d37dad79ce809ba9ee0603afcac26229: Status 404 returned error can't find the container with id 3d6fd9f62cc6e29b9f554d1e18447a05d37dad79ce809ba9ee0603afcac26229
Jul 20 10:43:22.438122 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.438025    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" event={"ID":"ae2237c8-af79-49c7-ad6a-0c5bea660d8d","Type":"ContainerStarted","Data":"30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd"}
Jul 20 10:43:22.438122 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.438085    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" event={"ID":"ae2237c8-af79-49c7-ad6a-0c5bea660d8d","Type":"ContainerStarted","Data":"3d6fd9f62cc6e29b9f554d1e18447a05d37dad79ce809ba9ee0603afcac26229"}
Jul 20 10:43:22.485849 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.485811    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:22.631474 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:22.631440    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"]
Jul 20 10:43:22.634507 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:43:22.634473    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd160edff_b38f_42cc_9caf_6a75b62b7470.slice/crio-7a600772181c7acf6db4338ecc46c58437a9fbc81e981b4c306f8472545bfaa8 WatchSource:0}: Error finding container 7a600772181c7acf6db4338ecc46c58437a9fbc81e981b4c306f8472545bfaa8: Status 404 returned error can't find the container with id 7a600772181c7acf6db4338ecc46c58437a9fbc81e981b4c306f8472545bfaa8
Jul 20 10:43:23.445245 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:23.445206    2572 generic.go:356] "Generic (PLEG): container finished" podID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerID="63c960afa2bdc7337dabb5071f2b91fac2df5a15ecbba25b1edd5f0116e7cea5" exitCode=0
Jul 20 10:43:23.445742 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:23.445244    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" event={"ID":"d160edff-b38f-42cc-9caf-6a75b62b7470","Type":"ContainerDied","Data":"63c960afa2bdc7337dabb5071f2b91fac2df5a15ecbba25b1edd5f0116e7cea5"}
Jul 20 10:43:23.445742 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:23.445303    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" event={"ID":"d160edff-b38f-42cc-9caf-6a75b62b7470","Type":"ContainerStarted","Data":"7a600772181c7acf6db4338ecc46c58437a9fbc81e981b4c306f8472545bfaa8"}
Jul 20 10:43:24.452651 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:24.452610    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" event={"ID":"d160edff-b38f-42cc-9caf-6a75b62b7470","Type":"ContainerStarted","Data":"12b437fde64d21a275b4ab4b07be37694b201d631436ee8971346334ee0787a0"}
Jul 20 10:43:24.453378 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:24.452771    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:43:24.496134 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:24.496059    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" podStartSLOduration=1.8599885889999999 podStartE2EDuration="2.496035936s" totalImagesPullingTime="0s" totalInitContainerRuntime="636.047347ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:43:24.49334851 +0000 UTC m=+2784.506450405" watchObservedRunningTime="2026-07-20 10:43:24.496035936 +0000 UTC m=+2784.509137839"
Jul 20 10:43:26.463303 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:26.463268    2572 generic.go:356] "Generic (PLEG): container finished" podID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerID="30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd" exitCode=0
Jul 20 10:43:26.463715 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:26.463343    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" event={"ID":"ae2237c8-af79-49c7-ad6a-0c5bea660d8d","Type":"ContainerDied","Data":"30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd"}
Jul 20 10:43:27.470213 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:27.470172    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" event={"ID":"ae2237c8-af79-49c7-ad6a-0c5bea660d8d","Type":"ContainerStarted","Data":"aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286"}
Jul 20 10:43:27.500379 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:27.500324    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podStartSLOduration=3.212864215 podStartE2EDuration="6.500309623s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.287445408s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:43:27.499774512 +0000 UTC m=+2787.512876428" watchObservedRunningTime="2026-07-20 10:43:27.500309623 +0000 UTC m=+2787.513411519"
Jul 20 10:43:28.184625 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:28.184583    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:32.184639 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:32.184597    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:43:32.186184 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:32.186157    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:43:42.185285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:42.185244    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:43:52.185263 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:52.185215    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:43:55.460025 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:43:55.459996    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:44:02.184768 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:44:02.184717    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:44:12.185201 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:44:12.185151    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:44:22.184904 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:44:22.184855    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:44:32.185635 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:44:32.185552    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:44:42.185480 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:44:42.185427    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:44:52.184913 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:44:52.184808    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:45:02.185023 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:02.184964    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:45:12.185020 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:12.184967    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:45:22.194786 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:22.194757    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:45:22.203010 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:22.202983    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:45:23.847717 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:23.847684    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"]
Jul 20 10:45:23.852846 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:23.852821    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"]
Jul 20 10:45:25.460041 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:25.460002    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:45:35.459904 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:35.459862    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:45:38.867612 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:38.867572    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerName="main" containerID="cri-o://12b437fde64d21a275b4ab4b07be37694b201d631436ee8971346334ee0787a0" gracePeriod=45
Jul 20 10:45:38.984615 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:38.984541    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" containerID="cri-o://aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286" gracePeriod=45
Jul 20 10:45:39.028637 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.028588    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" event={"ID":"d160edff-b38f-42cc-9caf-6a75b62b7470","Type":"ContainerDied","Data":"12b437fde64d21a275b4ab4b07be37694b201d631436ee8971346334ee0787a0"}
Jul 20 10:45:39.028815 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.028612    2572 generic.go:356] "Generic (PLEG): container finished" podID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerID="12b437fde64d21a275b4ab4b07be37694b201d631436ee8971346334ee0787a0" exitCode=0
Jul 20 10:45:39.112883 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.112861    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:45:39.196320 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.196230    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 20 10:45:39.242135 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.242108    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-tokenizer-uds\") pod \"d160edff-b38f-42cc-9caf-6a75b62b7470\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") "
Jul 20 10:45:39.242285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.242157    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d160edff-b38f-42cc-9caf-6a75b62b7470-tls-certs\") pod \"d160edff-b38f-42cc-9caf-6a75b62b7470\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") "
Jul 20 10:45:39.242285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.242208    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-kserve-provision-location\") pod \"d160edff-b38f-42cc-9caf-6a75b62b7470\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") "
Jul 20 10:45:39.242285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.242239    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvpjb\" (UniqueName: \"kubernetes.io/projected/d160edff-b38f-42cc-9caf-6a75b62b7470-kube-api-access-jvpjb\") pod \"d160edff-b38f-42cc-9caf-6a75b62b7470\" (UID: \"d160edff-b38f-42cc-9caf-6a75b62b7470\") "
Jul 20 10:45:39.242496 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.242451    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d160edff-b38f-42cc-9caf-6a75b62b7470" (UID: "d160edff-b38f-42cc-9caf-6a75b62b7470"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:39.242934 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.242908    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d160edff-b38f-42cc-9caf-6a75b62b7470" (UID: "d160edff-b38f-42cc-9caf-6a75b62b7470"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:39.244242 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.244221    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d160edff-b38f-42cc-9caf-6a75b62b7470-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d160edff-b38f-42cc-9caf-6a75b62b7470" (UID: "d160edff-b38f-42cc-9caf-6a75b62b7470"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:45:39.244300 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.244278    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d160edff-b38f-42cc-9caf-6a75b62b7470-kube-api-access-jvpjb" (OuterVolumeSpecName: "kube-api-access-jvpjb") pod "d160edff-b38f-42cc-9caf-6a75b62b7470" (UID: "d160edff-b38f-42cc-9caf-6a75b62b7470"). InnerVolumeSpecName "kube-api-access-jvpjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:45:39.343349 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.343309    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.343349 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.343344    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d160edff-b38f-42cc-9caf-6a75b62b7470-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.343349 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.343353    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d160edff-b38f-42cc-9caf-6a75b62b7470-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.343609 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.343363    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpjb\" (UniqueName: \"kubernetes.io/projected/d160edff-b38f-42cc-9caf-6a75b62b7470-kube-api-access-jvpjb\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.719512 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.719468    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:45:39.847421 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.847326    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tmp-dir\") pod \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") "
Jul 20 10:45:39.847421 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.847385    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kserve-provision-location\") pod \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") "
Jul 20 10:45:39.847421 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.847420    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-home\") pod \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") "
Jul 20 10:45:39.847744 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.847468    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-model-cache\") pod \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") "
Jul 20 10:45:39.847744 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.847500    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tls-certs\") pod \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") "
Jul 20 10:45:39.847744 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.847531    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcqt7\" (UniqueName: \"kubernetes.io/projected/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kube-api-access-mcqt7\") pod \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") "
Jul 20 10:45:39.847744 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.847580    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-dshm\") pod \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\" (UID: \"ae2237c8-af79-49c7-ad6a-0c5bea660d8d\") "
Jul 20 10:45:39.847942 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.847785    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae2237c8-af79-49c7-ad6a-0c5bea660d8d" (UID: "ae2237c8-af79-49c7-ad6a-0c5bea660d8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:39.848144 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.848120    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-home" (OuterVolumeSpecName: "home") pod "ae2237c8-af79-49c7-ad6a-0c5bea660d8d" (UID: "ae2237c8-af79-49c7-ad6a-0c5bea660d8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:39.850277 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.850249    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae2237c8-af79-49c7-ad6a-0c5bea660d8d" (UID: "ae2237c8-af79-49c7-ad6a-0c5bea660d8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:45:39.850277 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.850258    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kube-api-access-mcqt7" (OuterVolumeSpecName: "kube-api-access-mcqt7") pod "ae2237c8-af79-49c7-ad6a-0c5bea660d8d" (UID: "ae2237c8-af79-49c7-ad6a-0c5bea660d8d"). InnerVolumeSpecName "kube-api-access-mcqt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:45:39.850436 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.850351    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-dshm" (OuterVolumeSpecName: "dshm") pod "ae2237c8-af79-49c7-ad6a-0c5bea660d8d" (UID: "ae2237c8-af79-49c7-ad6a-0c5bea660d8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:39.858739 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.858715    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae2237c8-af79-49c7-ad6a-0c5bea660d8d" (UID: "ae2237c8-af79-49c7-ad6a-0c5bea660d8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:39.916548 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.916494    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae2237c8-af79-49c7-ad6a-0c5bea660d8d" (UID: "ae2237c8-af79-49c7-ad6a-0c5bea660d8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:39.948892 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.948860    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.948892 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.948890    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.949022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.948899    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.949022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.948908    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.949022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.948917    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqt7\" (UniqueName: \"kubernetes.io/projected/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-kube-api-access-mcqt7\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.949022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.948924    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:39.949022 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:39.948953    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2237c8-af79-49c7-ad6a-0c5bea660d8d-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:45:40.034251 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.034219    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj" event={"ID":"d160edff-b38f-42cc-9caf-6a75b62b7470","Type":"ContainerDied","Data":"7a600772181c7acf6db4338ecc46c58437a9fbc81e981b4c306f8472545bfaa8"}
Jul 20 10:45:40.034411 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.034285    2572 scope.go:117] "RemoveContainer" containerID="12b437fde64d21a275b4ab4b07be37694b201d631436ee8971346334ee0787a0"
Jul 20 10:45:40.034411 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.034228    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"
Jul 20 10:45:40.035801 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.035767    2572 generic.go:356] "Generic (PLEG): container finished" podID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerID="aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286" exitCode=0
Jul 20 10:45:40.035921 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.035809    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" event={"ID":"ae2237c8-af79-49c7-ad6a-0c5bea660d8d","Type":"ContainerDied","Data":"aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286"}
Jul 20 10:45:40.035921 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.035826    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"
Jul 20 10:45:40.035921 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.035846    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t" event={"ID":"ae2237c8-af79-49c7-ad6a-0c5bea660d8d","Type":"ContainerDied","Data":"3d6fd9f62cc6e29b9f554d1e18447a05d37dad79ce809ba9ee0603afcac26229"}
Jul 20 10:45:40.043769 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.043752    2572 scope.go:117] "RemoveContainer" containerID="63c960afa2bdc7337dabb5071f2b91fac2df5a15ecbba25b1edd5f0116e7cea5"
Jul 20 10:45:40.052509 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.052450    2572 scope.go:117] "RemoveContainer" containerID="aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286"
Jul 20 10:45:40.063587 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.063548    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"]
Jul 20 10:45:40.063998 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.063983    2572 scope.go:117] "RemoveContainer" containerID="30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd"
Jul 20 10:45:40.068344 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.068324    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c8c764d4d-znngj"]
Jul 20 10:45:40.071761 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.071746    2572 scope.go:117] "RemoveContainer" containerID="aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286"
Jul 20 10:45:40.072027 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:45:40.072007    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286\": container with ID starting with aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286 not found: ID does not exist" containerID="aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286"
Jul 20 10:45:40.072113 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.072049    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286"} err="failed to get container status \"aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286\": rpc error: code = NotFound desc = could not find container \"aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286\": container with ID starting with aae3c570348c975241cf28f48acb7c50ed55d8415ac61a0ccfae94f171653286 not found: ID does not exist"
Jul 20 10:45:40.072113 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.072064    2572 scope.go:117] "RemoveContainer" containerID="30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd"
Jul 20 10:45:40.072327 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:45:40.072310    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd\": container with ID starting with 30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd not found: ID does not exist" containerID="30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd"
Jul 20 10:45:40.072372 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.072333    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd"} err="failed to get container status \"30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd\": rpc error: code = NotFound desc = could not find container \"30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd\": container with ID starting with 30e35c3ed85fe975c9e530f775755baf5eddd014f0c2be9a7c5d38cb43548fbd not found: ID does not exist"
Jul 20 10:45:40.083855 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.083833    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"]
Jul 20 10:45:40.086750 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.086729    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd9f9b479-z549t"]
Jul 20 10:45:40.520078 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.520043    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" path="/var/lib/kubelet/pods/ae2237c8-af79-49c7-ad6a-0c5bea660d8d/volumes"
Jul 20 10:45:40.520610 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:40.520594    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" path="/var/lib/kubelet/pods/d160edff-b38f-42cc-9caf-6a75b62b7470/volumes"
Jul 20 10:45:53.909203 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909166    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"]
Jul 20 10:45:53.909604 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909588    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main"
Jul 20 10:45:53.909659 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909607    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main"
Jul 20 10:45:53.909659 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909621    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerName="storage-initializer"
Jul 20 10:45:53.909659 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909631    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerName="storage-initializer"
Jul 20 10:45:53.909659 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909650    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerName="main"
Jul 20 10:45:53.909659 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909659    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerName="main"
Jul 20 10:45:53.909814 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909684    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="storage-initializer"
Jul 20 10:45:53.909814 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909692    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="storage-initializer"
Jul 20 10:45:53.909814 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909765    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d160edff-b38f-42cc-9caf-6a75b62b7470" containerName="main"
Jul 20 10:45:53.909814 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.909775    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2237c8-af79-49c7-ad6a-0c5bea660d8d" containerName="main"
Jul 20 10:45:53.914356 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.914339    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:53.918224 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.918203    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 20 10:45:53.918534 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.918519    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-b7n62\""
Jul 20 10:45:53.918796 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.918780    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:45:53.918980 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.918965    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 20 10:45:53.919865 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.919843    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 20 10:45:53.920298 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.919855    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 20 10:45:53.922423 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.922403    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"]
Jul 20 10:45:53.952809 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.952770    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-home\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:53.952809 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.952817    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-model-cache\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:53.953039 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.952836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-tmp-dir\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:53.953039 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.952898    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-dshm\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:53.953039 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.952940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrz8\" (UniqueName: \"kubernetes.io/projected/469efb20-5f09-4944-84bd-fb2c2e60fc22-kube-api-access-9nrz8\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:53.953039 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.952961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/469efb20-5f09-4944-84bd-fb2c2e60fc22-cabundle-cert\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:53.953039 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.952990    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:53.953039 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:53.953008    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469efb20-5f09-4944-84bd-fb2c2e60fc22-tls-certs\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.053876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.053839    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.053876 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.053882    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469efb20-5f09-4944-84bd-fb2c2e60fc22-tls-certs\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054132 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.053910    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-home\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054132 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.053933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-model-cache\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054132 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.053950    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-tmp-dir\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054132 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.053974    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-dshm\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054132 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.053997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrz8\" (UniqueName: \"kubernetes.io/projected/469efb20-5f09-4944-84bd-fb2c2e60fc22-kube-api-access-9nrz8\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054132 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.054025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/469efb20-5f09-4944-84bd-fb2c2e60fc22-cabundle-cert\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054450 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.054346    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054548 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.054525    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-tmp-dir\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054633 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.054546    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-home\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054633 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.054616    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-model-cache\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.054803 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.054786    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/469efb20-5f09-4944-84bd-fb2c2e60fc22-cabundle-cert\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.056368 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.056349    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-dshm\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.056651 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.056629    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469efb20-5f09-4944-84bd-fb2c2e60fc22-tls-certs\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.062143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.062122    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrz8\" (UniqueName: \"kubernetes.io/projected/469efb20-5f09-4944-84bd-fb2c2e60fc22-kube-api-access-9nrz8\") pod \"lora-single-adapter-test-kserve-697f4bf8b6-zvg24\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.225791 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.225757    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:45:54.364983 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:54.364958    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"]
Jul 20 10:45:54.366815 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:45:54.366787    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469efb20_5f09_4944_84bd_fb2c2e60fc22.slice/crio-bcb5762c8e1c6ddf6f1c345db192fdd56015eed528a77afeab48b1741cf8423c WatchSource:0}: Error finding container bcb5762c8e1c6ddf6f1c345db192fdd56015eed528a77afeab48b1741cf8423c: Status 404 returned error can't find the container with id bcb5762c8e1c6ddf6f1c345db192fdd56015eed528a77afeab48b1741cf8423c
Jul 20 10:45:55.095845 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:55.095799    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" event={"ID":"469efb20-5f09-4944-84bd-fb2c2e60fc22","Type":"ContainerStarted","Data":"cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc"}
Jul 20 10:45:55.096230 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:55.095854    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" event={"ID":"469efb20-5f09-4944-84bd-fb2c2e60fc22","Type":"ContainerStarted","Data":"bcb5762c8e1c6ddf6f1c345db192fdd56015eed528a77afeab48b1741cf8423c"}
Jul 20 10:45:58.111683 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:58.111650    2572 generic.go:356] "Generic (PLEG): container finished" podID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerID="cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc" exitCode=0
Jul 20 10:45:58.112063 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:58.111714    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" event={"ID":"469efb20-5f09-4944-84bd-fb2c2e60fc22","Type":"ContainerDied","Data":"cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc"}
Jul 20 10:45:59.118492 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:59.118454    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" event={"ID":"469efb20-5f09-4944-84bd-fb2c2e60fc22","Type":"ContainerStarted","Data":"6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e"}
Jul 20 10:45:59.143247 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:59.143192    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podStartSLOduration=2.675395646 podStartE2EDuration="6.143176438s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.467780792s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:45:59.14021689 +0000 UTC m=+2939.153318783" watchObservedRunningTime="2026-07-20 10:45:59.143176438 +0000 UTC m=+2939.156278333"
Jul 20 10:45:59.226800 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:45:59.226771    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:46:04.226116 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:46:04.226068    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:46:04.227697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:46:04.227669    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 20 10:46:14.227217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:46:14.227176    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 20 10:46:24.226791 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:46:24.226699    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 20 10:46:34.227201 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:46:34.227155    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 20 10:46:44.226921 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:46:44.226867    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 20 10:46:54.226352 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:46:54.226300    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 20 10:47:00.610566 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:47:00.610525    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:47:00.625458 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:47:00.625435    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:47:04.226644 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:47:04.226596    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 20 10:47:14.237024 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:47:14.236983    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:47:14.244743 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:47:14.244716    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:47:49.661839 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:47:49.661805    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-697f4bf8b6-zvg24_469efb20-5f09-4944-84bd-fb2c2e60fc22/storage-initializer/0.log"
Jul 20 10:47:49.671226 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:47:49.671200    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-697f4bf8b6-zvg24_469efb20-5f09-4944-84bd-fb2c2e60fc22/main/0.log"
Jul 20 10:47:50.739106 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:47:50.739068    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"]
Jul 20 10:48:05.762108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:05.762055    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" containerID="cri-o://6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e" gracePeriod=45
Jul 20 10:48:06.237811 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.237771    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 20 10:48:06.412841 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.412816    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:48:06.429458 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.429435    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-home\") pod \"469efb20-5f09-4944-84bd-fb2c2e60fc22\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") "
Jul 20 10:48:06.429572 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.429468    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/469efb20-5f09-4944-84bd-fb2c2e60fc22-cabundle-cert\") pod \"469efb20-5f09-4944-84bd-fb2c2e60fc22\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") "
Jul 20 10:48:06.429572 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.429496    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-kserve-provision-location\") pod \"469efb20-5f09-4944-84bd-fb2c2e60fc22\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") "
Jul 20 10:48:06.429572 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.429521    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nrz8\" (UniqueName: \"kubernetes.io/projected/469efb20-5f09-4944-84bd-fb2c2e60fc22-kube-api-access-9nrz8\") pod \"469efb20-5f09-4944-84bd-fb2c2e60fc22\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") "
Jul 20 10:48:06.429740 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.429573    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469efb20-5f09-4944-84bd-fb2c2e60fc22-tls-certs\") pod \"469efb20-5f09-4944-84bd-fb2c2e60fc22\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") "
Jul 20 10:48:06.429740 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.429607    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-model-cache\") pod \"469efb20-5f09-4944-84bd-fb2c2e60fc22\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") "
Jul 20 10:48:06.429740 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.429702    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-tmp-dir\") pod \"469efb20-5f09-4944-84bd-fb2c2e60fc22\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") "
Jul 20 10:48:06.430325 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.430213    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-home" (OuterVolumeSpecName: "home") pod "469efb20-5f09-4944-84bd-fb2c2e60fc22" (UID: "469efb20-5f09-4944-84bd-fb2c2e60fc22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:48:06.430325 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.430272    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469efb20-5f09-4944-84bd-fb2c2e60fc22-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "469efb20-5f09-4944-84bd-fb2c2e60fc22" (UID: "469efb20-5f09-4944-84bd-fb2c2e60fc22"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:48:06.441608 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.430182    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-model-cache" (OuterVolumeSpecName: "model-cache") pod "469efb20-5f09-4944-84bd-fb2c2e60fc22" (UID: "469efb20-5f09-4944-84bd-fb2c2e60fc22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:48:06.441608 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.430679    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-dshm\") pod \"469efb20-5f09-4944-84bd-fb2c2e60fc22\" (UID: \"469efb20-5f09-4944-84bd-fb2c2e60fc22\") "
Jul 20 10:48:06.441608 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.431401    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:48:06.441608 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.431422    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/469efb20-5f09-4944-84bd-fb2c2e60fc22-cabundle-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:48:06.441608 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.431437    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:48:06.442894 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.442840    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-dshm" (OuterVolumeSpecName: "dshm") pod "469efb20-5f09-4944-84bd-fb2c2e60fc22" (UID: "469efb20-5f09-4944-84bd-fb2c2e60fc22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:48:06.444325 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.444277    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469efb20-5f09-4944-84bd-fb2c2e60fc22-kube-api-access-9nrz8" (OuterVolumeSpecName: "kube-api-access-9nrz8") pod "469efb20-5f09-4944-84bd-fb2c2e60fc22" (UID: "469efb20-5f09-4944-84bd-fb2c2e60fc22"). InnerVolumeSpecName "kube-api-access-9nrz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:48:06.444468 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.444365    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469efb20-5f09-4944-84bd-fb2c2e60fc22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "469efb20-5f09-4944-84bd-fb2c2e60fc22" (UID: "469efb20-5f09-4944-84bd-fb2c2e60fc22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:48:06.449291 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.449260    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "469efb20-5f09-4944-84bd-fb2c2e60fc22" (UID: "469efb20-5f09-4944-84bd-fb2c2e60fc22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:48:06.506906 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.506863    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "469efb20-5f09-4944-84bd-fb2c2e60fc22" (UID: "469efb20-5f09-4944-84bd-fb2c2e60fc22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:48:06.532536 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.532510    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:48:06.532536 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.532536    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nrz8\" (UniqueName: \"kubernetes.io/projected/469efb20-5f09-4944-84bd-fb2c2e60fc22-kube-api-access-9nrz8\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:48:06.532687 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.532546    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469efb20-5f09-4944-84bd-fb2c2e60fc22-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:48:06.532687 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.532566    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:48:06.532687 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.532574    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469efb20-5f09-4944-84bd-fb2c2e60fc22-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:48:06.660981 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.660939    2572 generic.go:356] "Generic (PLEG): container finished" podID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerID="6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e" exitCode=0
Jul 20 10:48:06.661177 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.660991    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" event={"ID":"469efb20-5f09-4944-84bd-fb2c2e60fc22","Type":"ContainerDied","Data":"6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e"}
Jul 20 10:48:06.661177 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.661013    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"
Jul 20 10:48:06.661177 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.661052    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24" event={"ID":"469efb20-5f09-4944-84bd-fb2c2e60fc22","Type":"ContainerDied","Data":"bcb5762c8e1c6ddf6f1c345db192fdd56015eed528a77afeab48b1741cf8423c"}
Jul 20 10:48:06.661177 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.661077    2572 scope.go:117] "RemoveContainer" containerID="6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e"
Jul 20 10:48:06.669424 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.669406    2572 scope.go:117] "RemoveContainer" containerID="cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc"
Jul 20 10:48:06.677302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.677267    2572 scope.go:117] "RemoveContainer" containerID="6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e"
Jul 20 10:48:06.677663 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:48:06.677628    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e\": container with ID starting with 6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e not found: ID does not exist" containerID="6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e"
Jul 20 10:48:06.677748 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.677679    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e"} err="failed to get container status \"6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e\": rpc error: code = NotFound desc = could not find container \"6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e\": container with ID starting with 6814f5b51dc702935f86223cc8271b551e734743f95e0937fc6b7ee76333a69e not found: ID does not exist"
Jul 20 10:48:06.677748 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.677708    2572 scope.go:117] "RemoveContainer" containerID="cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc"
Jul 20 10:48:06.678038 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:48:06.677981    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc\": container with ID starting with cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc not found: ID does not exist" containerID="cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc"
Jul 20 10:48:06.678038 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.678016    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc"} err="failed to get container status \"cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc\": rpc error: code = NotFound desc = could not find container \"cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc\": container with ID starting with cb3c8961574b69ccc2561ecf3b029c4eb08b81a51518f3968d99ec478a3a03fc not found: ID does not exist"
Jul 20 10:48:06.679376 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.679355    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"]
Jul 20 10:48:06.682721 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:06.682699    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-697f4bf8b6-zvg24"]
Jul 20 10:48:08.520178 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:08.520146    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" path="/var/lib/kubelet/pods/469efb20-5f09-4944-84bd-fb2c2e60fc22/volumes"
Jul 20 10:48:28.410693 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.410656    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"]
Jul 20 10:48:28.411070 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.410936    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="storage-initializer"
Jul 20 10:48:28.411070 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.410947    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="storage-initializer"
Jul 20 10:48:28.411070 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.410955    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main"
Jul 20 10:48:28.411070 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.410961    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main"
Jul 20 10:48:28.411070 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.411019    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="469efb20-5f09-4944-84bd-fb2c2e60fc22" containerName="main"
Jul 20 10:48:28.414123 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.414106    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.416678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.416654    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 20 10:48:28.416828 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.416759    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:48:28.416828 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.416807    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 20 10:48:28.416935 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.416871    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 20 10:48:28.416981 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.416965    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-vlvlm\""
Jul 20 10:48:28.417108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.417091    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 20 10:48:28.424030 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.424008    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"]
Jul 20 10:48:28.509051 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.509013    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903653d6-f762-4b24-bee9-40a9eb77c24b-tls-certs\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.509269 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.509062    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/903653d6-f762-4b24-bee9-40a9eb77c24b-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.509269 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.509135    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzxv\" (UniqueName: \"kubernetes.io/projected/903653d6-f762-4b24-bee9-40a9eb77c24b-kube-api-access-nwzxv\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.509269 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.509196    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.509741 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.509223    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.509891 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.509837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-dshm\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.509958 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.509890    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-home\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.509958 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.509951    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-model-cache\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611224 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611178    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/903653d6-f762-4b24-bee9-40a9eb77c24b-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611224 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzxv\" (UniqueName: \"kubernetes.io/projected/903653d6-f762-4b24-bee9-40a9eb77c24b-kube-api-access-nwzxv\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611465 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611465 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611465 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611313    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-dshm\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611465 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-home\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611465 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-model-cache\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611465 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903653d6-f762-4b24-bee9-40a9eb77c24b-tls-certs\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.611817 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.611773    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.612063 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.612042    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-model-cache\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.612137 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.612044    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/903653d6-f762-4b24-bee9-40a9eb77c24b-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.612137 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.612044    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-home\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.612137 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.612116    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.613787 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.613759    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-dshm\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.613981 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.613965    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903653d6-f762-4b24-bee9-40a9eb77c24b-tls-certs\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.619009 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.618989    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzxv\" (UniqueName: \"kubernetes.io/projected/903653d6-f762-4b24-bee9-40a9eb77c24b-kube-api-access-nwzxv\") pod \"lora-multiple-adapters-test-kserve-54b959495c-45jzn\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.725105 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.725062    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:28.866054 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:28.866021    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"]
Jul 20 10:48:28.869520 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:48:28.869492    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903653d6_f762_4b24_bee9_40a9eb77c24b.slice/crio-328e32b2ec526b273f45eac4f29cd2f1f3ca99f41e219c10a52ccf5d4bb15b71 WatchSource:0}: Error finding container 328e32b2ec526b273f45eac4f29cd2f1f3ca99f41e219c10a52ccf5d4bb15b71: Status 404 returned error can't find the container with id 328e32b2ec526b273f45eac4f29cd2f1f3ca99f41e219c10a52ccf5d4bb15b71
Jul 20 10:48:29.753640 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:29.753598    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" event={"ID":"903653d6-f762-4b24-bee9-40a9eb77c24b","Type":"ContainerStarted","Data":"303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715"}
Jul 20 10:48:29.754033 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:29.753654    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" event={"ID":"903653d6-f762-4b24-bee9-40a9eb77c24b","Type":"ContainerStarted","Data":"328e32b2ec526b273f45eac4f29cd2f1f3ca99f41e219c10a52ccf5d4bb15b71"}
Jul 20 10:48:32.767688 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:32.767599    2572 generic.go:356] "Generic (PLEG): container finished" podID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerID="303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715" exitCode=0
Jul 20 10:48:32.767688 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:32.767652    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" event={"ID":"903653d6-f762-4b24-bee9-40a9eb77c24b","Type":"ContainerDied","Data":"303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715"}
Jul 20 10:48:33.773312 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:33.773279    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" event={"ID":"903653d6-f762-4b24-bee9-40a9eb77c24b","Type":"ContainerStarted","Data":"a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57"}
Jul 20 10:48:33.798631 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:33.798581    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podStartSLOduration=2.221065933 podStartE2EDuration="5.798524058s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.577458125s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:48:33.797828059 +0000 UTC m=+3093.810929969" watchObservedRunningTime="2026-07-20 10:48:33.798524058 +0000 UTC m=+3093.811625953"
Jul 20 10:48:34.726080 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:34.726041    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:38.725744 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:38.725713    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:48:38.727170 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:38.727139    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 20 10:48:48.725996 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:48.725948    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 20 10:48:58.725967 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:48:58.725925    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 20 10:49:08.725959 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:49:08.725919    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 20 10:49:18.726576 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:49:18.726523    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 20 10:49:28.725708 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:49:28.725667    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 20 10:49:38.726381 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:49:38.726336    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 20 10:49:48.735734 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:49:48.735701    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:49:48.743471 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:49:48.743446    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:50:35.070306 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:35.070273    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-54b959495c-45jzn_903653d6-f762-4b24-bee9-40a9eb77c24b/storage-initializer/0.log"
Jul 20 10:50:35.081549 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:35.081523    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-54b959495c-45jzn_903653d6-f762-4b24-bee9-40a9eb77c24b/main/0.log"
Jul 20 10:50:36.262953 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:36.262918    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"]
Jul 20 10:50:51.285919 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:51.285855    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" containerID="cri-o://a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57" gracePeriod=45
Jul 20 10:50:51.736478 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:51.736427    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 20 10:50:51.943600 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:51.943572    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:50:52.015898 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.015807    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-tmp-dir\") pod \"903653d6-f762-4b24-bee9-40a9eb77c24b\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") "
Jul 20 10:50:52.015898 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.015846    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwzxv\" (UniqueName: \"kubernetes.io/projected/903653d6-f762-4b24-bee9-40a9eb77c24b-kube-api-access-nwzxv\") pod \"903653d6-f762-4b24-bee9-40a9eb77c24b\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") "
Jul 20 10:50:52.015898 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.015864    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-model-cache\") pod \"903653d6-f762-4b24-bee9-40a9eb77c24b\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") "
Jul 20 10:50:52.015898 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.015900    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903653d6-f762-4b24-bee9-40a9eb77c24b-tls-certs\") pod \"903653d6-f762-4b24-bee9-40a9eb77c24b\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") "
Jul 20 10:50:52.016181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.016018    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-kserve-provision-location\") pod \"903653d6-f762-4b24-bee9-40a9eb77c24b\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") "
Jul 20 10:50:52.016181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.016068    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-dshm\") pod \"903653d6-f762-4b24-bee9-40a9eb77c24b\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") "
Jul 20 10:50:52.016181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.016104    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/903653d6-f762-4b24-bee9-40a9eb77c24b-cabundle-cert\") pod \"903653d6-f762-4b24-bee9-40a9eb77c24b\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") "
Jul 20 10:50:52.016181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.016148    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-home\") pod \"903653d6-f762-4b24-bee9-40a9eb77c24b\" (UID: \"903653d6-f762-4b24-bee9-40a9eb77c24b\") "
Jul 20 10:50:52.016181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.016153    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-model-cache" (OuterVolumeSpecName: "model-cache") pod "903653d6-f762-4b24-bee9-40a9eb77c24b" (UID: "903653d6-f762-4b24-bee9-40a9eb77c24b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:50:52.016444 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.016358    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:50:52.016580 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.016537    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903653d6-f762-4b24-bee9-40a9eb77c24b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "903653d6-f762-4b24-bee9-40a9eb77c24b" (UID: "903653d6-f762-4b24-bee9-40a9eb77c24b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:50:52.016737 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.016714    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-home" (OuterVolumeSpecName: "home") pod "903653d6-f762-4b24-bee9-40a9eb77c24b" (UID: "903653d6-f762-4b24-bee9-40a9eb77c24b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:50:52.018325 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.018293    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-dshm" (OuterVolumeSpecName: "dshm") pod "903653d6-f762-4b24-bee9-40a9eb77c24b" (UID: "903653d6-f762-4b24-bee9-40a9eb77c24b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:50:52.018452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.018351    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903653d6-f762-4b24-bee9-40a9eb77c24b-kube-api-access-nwzxv" (OuterVolumeSpecName: "kube-api-access-nwzxv") pod "903653d6-f762-4b24-bee9-40a9eb77c24b" (UID: "903653d6-f762-4b24-bee9-40a9eb77c24b"). InnerVolumeSpecName "kube-api-access-nwzxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:50:52.018452 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.018411    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903653d6-f762-4b24-bee9-40a9eb77c24b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "903653d6-f762-4b24-bee9-40a9eb77c24b" (UID: "903653d6-f762-4b24-bee9-40a9eb77c24b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:50:52.028529 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.028500    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "903653d6-f762-4b24-bee9-40a9eb77c24b" (UID: "903653d6-f762-4b24-bee9-40a9eb77c24b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:50:52.083181 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.083132    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "903653d6-f762-4b24-bee9-40a9eb77c24b" (UID: "903653d6-f762-4b24-bee9-40a9eb77c24b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:50:52.116883 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.116853    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:50:52.116883 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.116881    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/903653d6-f762-4b24-bee9-40a9eb77c24b-cabundle-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:50:52.117053 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.116891    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:50:52.117053 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.116900    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:50:52.117053 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.116908    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwzxv\" (UniqueName: \"kubernetes.io/projected/903653d6-f762-4b24-bee9-40a9eb77c24b-kube-api-access-nwzxv\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:50:52.117053 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.116917    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903653d6-f762-4b24-bee9-40a9eb77c24b-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:50:52.117053 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.116925    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/903653d6-f762-4b24-bee9-40a9eb77c24b-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:50:52.363249 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.363162    2572 generic.go:356] "Generic (PLEG): container finished" podID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerID="a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57" exitCode=0
Jul 20 10:50:52.363249 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.363213    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"
Jul 20 10:50:52.363249 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.363228    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" event={"ID":"903653d6-f762-4b24-bee9-40a9eb77c24b","Type":"ContainerDied","Data":"a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57"}
Jul 20 10:50:52.363824 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.363296    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn" event={"ID":"903653d6-f762-4b24-bee9-40a9eb77c24b","Type":"ContainerDied","Data":"328e32b2ec526b273f45eac4f29cd2f1f3ca99f41e219c10a52ccf5d4bb15b71"}
Jul 20 10:50:52.363824 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.363317    2572 scope.go:117] "RemoveContainer" containerID="a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57"
Jul 20 10:50:52.372542 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.372517    2572 scope.go:117] "RemoveContainer" containerID="303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715"
Jul 20 10:50:52.380458 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.380437    2572 scope.go:117] "RemoveContainer" containerID="a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57"
Jul 20 10:50:52.380749 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:50:52.380727    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57\": container with ID starting with a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57 not found: ID does not exist" containerID="a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57"
Jul 20 10:50:52.380828 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.380762    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57"} err="failed to get container status \"a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57\": rpc error: code = NotFound desc = could not find container \"a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57\": container with ID starting with a8b5c41ddfd23a7faee9430222b6b6a8aae08e74facb3096c55dff76f0b7fe57 not found: ID does not exist"
Jul 20 10:50:52.380828 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.380789    2572 scope.go:117] "RemoveContainer" containerID="303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715"
Jul 20 10:50:52.381053 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:50:52.381035    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715\": container with ID starting with 303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715 not found: ID does not exist" containerID="303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715"
Jul 20 10:50:52.381091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.381060    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715"} err="failed to get container status \"303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715\": rpc error: code = NotFound desc = could not find container \"303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715\": container with ID starting with 303d89e68c2e464e6d5daaa6f95fb99ce794df50b8ae15e1b19a69b0bf7c0715 not found: ID does not exist"
Jul 20 10:50:52.387005 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.386980    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"]
Jul 20 10:50:52.392582 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.392544    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-54b959495c-45jzn"]
Jul 20 10:50:52.519733 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:50:52.519699    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" path="/var/lib/kubelet/pods/903653d6-f762-4b24-bee9-40a9eb77c24b/volumes"
Jul 20 10:51:08.708397 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.708351    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"]
Jul 20 10:51:08.708886 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.708723    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="storage-initializer"
Jul 20 10:51:08.708886 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.708735    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="storage-initializer"
Jul 20 10:51:08.708886 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.708752    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main"
Jul 20 10:51:08.708886 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.708760    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main"
Jul 20 10:51:08.708886 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.708841    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="903653d6-f762-4b24-bee9-40a9eb77c24b" containerName="main"
Jul 20 10:51:08.713877 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.713855    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.716619 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.716592    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 20 10:51:08.716757 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.716707    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-9jpwk\""
Jul 20 10:51:08.716921 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.716899    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 20 10:51:08.717018 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.716961    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 20 10:51:08.717075 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.717033    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 20 10:51:08.720659 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.720636    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"]
Jul 20 10:51:08.747367 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.747337    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tmp-dir\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.747491 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.747376    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-home\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.747491 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.747401    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-dshm\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.747491 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.747478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4zsg\" (UniqueName: \"kubernetes.io/projected/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kube-api-access-s4zsg\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.747643 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.747505    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-model-cache\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.747643 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.747537    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kserve-provision-location\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.747643 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.747599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tls-certs\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.848830 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.848784    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kserve-provision-location\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849029 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.848849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tls-certs\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849029 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.848899    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tmp-dir\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849174 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.849034    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-home\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849174 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.849077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-dshm\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849174 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.849138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zsg\" (UniqueName: \"kubernetes.io/projected/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kube-api-access-s4zsg\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849174 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.849167    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-model-cache\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849365 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.849181    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kserve-provision-location\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849365 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.849255    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tmp-dir\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849441 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.849362    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-home\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.849489 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.849474    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-model-cache\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.851291 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.851276    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-dshm\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.851588 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.851545    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tls-certs\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:08.857134 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:08.857115    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zsg\" (UniqueName: \"kubernetes.io/projected/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kube-api-access-s4zsg\") pod \"tls-verification-test-kserve-688f745986-jn26t\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:09.024622 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:09.024509    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:09.158550 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:09.158520    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"]
Jul 20 10:51:09.163412 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:51:09.163382    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039c8e7d_7e9d_42c0_a778_20e3ec9ed8d9.slice/crio-eec6942528fca26eee27672d1a071d16dac79c42df823f63b9bab60b38e165c7 WatchSource:0}: Error finding container eec6942528fca26eee27672d1a071d16dac79c42df823f63b9bab60b38e165c7: Status 404 returned error can't find the container with id eec6942528fca26eee27672d1a071d16dac79c42df823f63b9bab60b38e165c7
Jul 20 10:51:09.432308 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:09.432205    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" event={"ID":"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9","Type":"ContainerStarted","Data":"0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957"}
Jul 20 10:51:09.432308 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:09.432280    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" event={"ID":"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9","Type":"ContainerStarted","Data":"eec6942528fca26eee27672d1a071d16dac79c42df823f63b9bab60b38e165c7"}
Jul 20 10:51:13.448928 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:13.448889    2572 generic.go:356] "Generic (PLEG): container finished" podID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerID="0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957" exitCode=0
Jul 20 10:51:13.449315 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:13.448944    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" event={"ID":"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9","Type":"ContainerDied","Data":"0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957"}
Jul 20 10:51:14.455445 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:14.455406    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" event={"ID":"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9","Type":"ContainerStarted","Data":"1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f"}
Jul 20 10:51:14.480306 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:14.480250    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podStartSLOduration=3.049978724 podStartE2EDuration="6.48023447s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.430255746s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:51:14.478541679 +0000 UTC m=+3254.491643576" watchObservedRunningTime="2026-07-20 10:51:14.48023447 +0000 UTC m=+3254.493336365"
Jul 20 10:51:15.024989 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:15.024952    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:19.025576 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:19.025517    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:51:19.027219 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:19.027190    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:51:29.025913 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:29.025864    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:51:39.025043 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:39.024997    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:51:49.025771 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:49.025725    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:51:59.025431 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:51:59.025386    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:52:00.635957 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:52:00.635930    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:52:00.651055 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:52:00.651025    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:52:09.025055 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:52:09.025008    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:52:19.025459 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:52:19.025414    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:52:29.025939 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:52:29.025894    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:52:39.025185 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:52:39.025142    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:52:49.024919 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:52:49.024874    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:52:59.025608 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:52:59.025519    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 20 10:53:09.035515 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:09.035484    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:53:09.043546 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:09.043515    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:53:14.077890 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:14.077854    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"]
Jul 20 10:53:29.101396 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.101346    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main" containerID="cri-o://1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f" gracePeriod=45
Jul 20 10:53:29.758451 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.758427    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:53:29.821374 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821334    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tmp-dir\") pod \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") "
Jul 20 10:53:29.821374 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821376    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kserve-provision-location\") pod \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") "
Jul 20 10:53:29.821623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821401    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-model-cache\") pod \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") "
Jul 20 10:53:29.821623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821424    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-home\") pod \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") "
Jul 20 10:53:29.821623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821465    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tls-certs\") pod \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") "
Jul 20 10:53:29.821623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821513    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4zsg\" (UniqueName: \"kubernetes.io/projected/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kube-api-access-s4zsg\") pod \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") "
Jul 20 10:53:29.821623 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821569    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-dshm\") pod \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\" (UID: \"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9\") "
Jul 20 10:53:29.821958 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821662    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" (UID: "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:53:29.821958 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.821790    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.822210 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.822178    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-home" (OuterVolumeSpecName: "home") pod "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" (UID: "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:53:29.824124 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.824086    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" (UID: "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:53:29.824303 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.824274    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kube-api-access-s4zsg" (OuterVolumeSpecName: "kube-api-access-s4zsg") pod "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" (UID: "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9"). InnerVolumeSpecName "kube-api-access-s4zsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:53:29.824303 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.824294    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-dshm" (OuterVolumeSpecName: "dshm") pod "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" (UID: "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:53:29.834019 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.833969    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" (UID: "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:53:29.887188 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.887135    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" (UID: "039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:53:29.928087 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.923344    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4zsg\" (UniqueName: \"kubernetes.io/projected/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kube-api-access-s4zsg\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.928087 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.923380    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.928087 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.923396    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.928087 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.923411    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.928087 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.923431    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:53:29.928087 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:29.923444    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:53:30.005311 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.005269    2572 generic.go:356] "Generic (PLEG): container finished" podID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerID="1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f" exitCode=0
Jul 20 10:53:30.005481 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.005332    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"
Jul 20 10:53:30.005481 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.005356    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" event={"ID":"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9","Type":"ContainerDied","Data":"1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f"}
Jul 20 10:53:30.005481 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.005435    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t" event={"ID":"039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9","Type":"ContainerDied","Data":"eec6942528fca26eee27672d1a071d16dac79c42df823f63b9bab60b38e165c7"}
Jul 20 10:53:30.005481 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.005460    2572 scope.go:117] "RemoveContainer" containerID="1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f"
Jul 20 10:53:30.014130 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.014113    2572 scope.go:117] "RemoveContainer" containerID="0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957"
Jul 20 10:53:30.021697 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.021678    2572 scope.go:117] "RemoveContainer" containerID="1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f"
Jul 20 10:53:30.021935 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:53:30.021917    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f\": container with ID starting with 1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f not found: ID does not exist" containerID="1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f"
Jul 20 10:53:30.021989 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.021945    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f"} err="failed to get container status \"1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f\": rpc error: code = NotFound desc = could not find container \"1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f\": container with ID starting with 1c46473653c55a0fc9b4da95ac0a198c83e635d3ff830e080af42463c9a9866f not found: ID does not exist"
Jul 20 10:53:30.021989 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.021966    2572 scope.go:117] "RemoveContainer" containerID="0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957"
Jul 20 10:53:30.022204 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:53:30.022186    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957\": container with ID starting with 0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957 not found: ID does not exist" containerID="0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957"
Jul 20 10:53:30.022259 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.022213    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957"} err="failed to get container status \"0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957\": rpc error: code = NotFound desc = could not find container \"0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957\": container with ID starting with 0a03fb64a9da972c514bb6434b78daf795b6861662e1b4e516828099bf10b957 not found: ID does not exist"
Jul 20 10:53:30.028826 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.028803    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"]
Jul 20 10:53:30.032609 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.032588    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-688f745986-jn26t"]
Jul 20 10:53:30.519347 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:30.519317    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" path="/var/lib/kubelet/pods/039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9/volumes"
Jul 20 10:53:39.432534 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.432501    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"]
Jul 20 10:53:39.432930 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.432809    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="storage-initializer"
Jul 20 10:53:39.432930 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.432821    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="storage-initializer"
Jul 20 10:53:39.432930 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.432840    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main"
Jul 20 10:53:39.432930 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.432845    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main"
Jul 20 10:53:39.432930 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.432896    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="039c8e7d-7e9d-42c0-a778-20e3ec9ed8d9" containerName="main"
Jul 20 10:53:39.437774 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.437754    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.440455 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.440431    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 20 10:53:39.440590 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.440431    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 20 10:53:39.440590 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.440490    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 20 10:53:39.440696 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.440601    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-t9s4k\""
Jul 20 10:53:39.440696 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.440683    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 20 10:53:39.447313 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.447291    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"]
Jul 20 10:53:39.602476 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.602437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-tmp-dir\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.602476 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.602477    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.602730 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.602504    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwpt\" (UniqueName: \"kubernetes.io/projected/5b507d3d-e739-40f7-a447-d2759db56379-kube-api-access-pzwpt\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.602730 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.602619    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-home\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.602803 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.602733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-model-cache\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.602803 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.602764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-dshm\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.602803 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.602792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b507d3d-e739-40f7-a447-d2759db56379-tls-certs\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.703820 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.703787    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-dshm\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.703820 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.703822    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b507d3d-e739-40f7-a447-d2759db56379-tls-certs\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704058 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.703850    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-tmp-dir\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704058 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.703868    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704058 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.703905    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwpt\" (UniqueName: \"kubernetes.io/projected/5b507d3d-e739-40f7-a447-d2759db56379-kube-api-access-pzwpt\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704058 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.703932    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-home\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704058 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.703998    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-model-cache\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704316 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.704286    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704438 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.704382    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-home\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704622 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.704588    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-model-cache\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.704719 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.704675    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-tmp-dir\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.706158 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.706138    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-dshm\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.706793 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.706767    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b507d3d-e739-40f7-a447-d2759db56379-tls-certs\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.714696 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.714665    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwpt\" (UniqueName: \"kubernetes.io/projected/5b507d3d-e739-40f7-a447-d2759db56379-kube-api-access-pzwpt\") pod \"prestop-hook-test-kserve-6f4448865d-4668s\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.749144 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.749106    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:39.883650 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:39.883625    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"]
Jul 20 10:53:39.885821 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:53:39.885797    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b507d3d_e739_40f7_a447_d2759db56379.slice/crio-97cda0c7bf135e5ac3241545827b800f69cb59fcb8bae0144d69cfa602d4df60 WatchSource:0}: Error finding container 97cda0c7bf135e5ac3241545827b800f69cb59fcb8bae0144d69cfa602d4df60: Status 404 returned error can't find the container with id 97cda0c7bf135e5ac3241545827b800f69cb59fcb8bae0144d69cfa602d4df60
Jul 20 10:53:40.048525 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:40.048419    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" event={"ID":"5b507d3d-e739-40f7-a447-d2759db56379","Type":"ContainerStarted","Data":"ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb"}
Jul 20 10:53:40.048525 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:40.048500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" event={"ID":"5b507d3d-e739-40f7-a447-d2759db56379","Type":"ContainerStarted","Data":"97cda0c7bf135e5ac3241545827b800f69cb59fcb8bae0144d69cfa602d4df60"}
Jul 20 10:53:44.066703 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:44.066664    2572 generic.go:356] "Generic (PLEG): container finished" podID="5b507d3d-e739-40f7-a447-d2759db56379" containerID="ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb" exitCode=0
Jul 20 10:53:44.067097 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:44.066730    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" event={"ID":"5b507d3d-e739-40f7-a447-d2759db56379","Type":"ContainerDied","Data":"ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb"}
Jul 20 10:53:45.072730 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:45.072695    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" event={"ID":"5b507d3d-e739-40f7-a447-d2759db56379","Type":"ContainerStarted","Data":"e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9"}
Jul 20 10:53:45.100910 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:45.100854    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podStartSLOduration=2.725208994 podStartE2EDuration="6.100838088s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.375629094s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:53:45.098785089 +0000 UTC m=+3405.111886987" watchObservedRunningTime="2026-07-20 10:53:45.100838088 +0000 UTC m=+3405.113939984"
Jul 20 10:53:45.750239 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:45.750197    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:49.749620 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:49.749579    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:53:49.751179 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:49.751147    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:53:59.750163 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:53:59.750065    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:54:09.750174 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:54:09.750127    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:54:19.750192 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:54:19.750143    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:54:29.750148 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:54:29.750103    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:54:39.749895 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:54:39.749840    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:54:49.749748 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:54:49.749704    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:54:59.749812 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:54:59.749767    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:55:09.750281 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:09.750236    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:55:19.750459 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:19.750407    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:55:29.749820 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:29.749769    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:55:39.759927 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:39.759890    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:55:39.767808 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:39.767784    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:55:39.934824 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:39.934788    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"]
Jul 20 10:55:39.958847 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:39.958818    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"]
Jul 20 10:55:39.964520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:39.964499    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:39.979525 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:39.979502    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"]
Jul 20 10:55:40.078258 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.078177    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-home\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.078258 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.078225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-dshm\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.078442 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.078280    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn967\" (UniqueName: \"kubernetes.io/projected/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kube-api-access-bn967\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.078442 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.078327    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-model-cache\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.078442 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.078361    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tmp-dir\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.078442 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.078381    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tls-certs\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.078442 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.078419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.179642 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.179606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.179807 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.179696    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-home\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.179807 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.179754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-dshm\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.179807 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.179781    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn967\" (UniqueName: \"kubernetes.io/projected/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kube-api-access-bn967\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.180113 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.179817    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-model-cache\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.180113 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.179853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tmp-dir\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.180113 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.179889    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tls-certs\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.180113 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.180040    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-home\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.180323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.180190    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.180323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.180231    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tmp-dir\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.180323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.180313    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-model-cache\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.182195 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.182171    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-dshm\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.182433 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.182416    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tls-certs\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.187587 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.187550    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn967\" (UniqueName: \"kubernetes.io/projected/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kube-api-access-bn967\") pod \"prestop-hook-test-kserve-6f4448865d-s9dcr\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.274801 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.274763    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:40.412755 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.412729    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"]
Jul 20 10:55:40.414598 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:55:40.414572    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc82f3c7_aec3_4d04_b1c5_1122a93e366f.slice/crio-beb48ecf3206497e1f0e27724c4dcf91094a1c6dce34efa722f8037a2b94892d WatchSource:0}: Error finding container beb48ecf3206497e1f0e27724c4dcf91094a1c6dce34efa722f8037a2b94892d: Status 404 returned error can't find the container with id beb48ecf3206497e1f0e27724c4dcf91094a1c6dce34efa722f8037a2b94892d
Jul 20 10:55:40.551995 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.551957    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" event={"ID":"fc82f3c7-aec3-4d04-b1c5-1122a93e366f","Type":"ContainerStarted","Data":"9c005517e8bcdbcb469e52d14289a3499174093f2d848d409bbe1c0cc1689f2b"}
Jul 20 10:55:40.552179 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:40.552020    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" event={"ID":"fc82f3c7-aec3-4d04-b1c5-1122a93e366f","Type":"ContainerStarted","Data":"beb48ecf3206497e1f0e27724c4dcf91094a1c6dce34efa722f8037a2b94892d"}
Jul 20 10:55:43.566546 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:43.566503    2572 generic.go:356] "Generic (PLEG): container finished" podID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerID="9c005517e8bcdbcb469e52d14289a3499174093f2d848d409bbe1c0cc1689f2b" exitCode=0
Jul 20 10:55:43.566942 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:43.566588    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" event={"ID":"fc82f3c7-aec3-4d04-b1c5-1122a93e366f","Type":"ContainerDied","Data":"9c005517e8bcdbcb469e52d14289a3499174093f2d848d409bbe1c0cc1689f2b"}
Jul 20 10:55:44.573076 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:44.573044    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" event={"ID":"fc82f3c7-aec3-4d04-b1c5-1122a93e366f","Type":"ContainerStarted","Data":"5b424bebcde218ab37e9fcafb17d77e16d07bbd2a1bad755621e38ffb291e77b"}
Jul 20 10:55:44.598072 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:44.598017    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" podStartSLOduration=2.65042894 podStartE2EDuration="5.598001577s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.947572637s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:55:44.595521765 +0000 UTC m=+3524.608623685" watchObservedRunningTime="2026-07-20 10:55:44.598001577 +0000 UTC m=+3524.611103522"
Jul 20 10:55:45.275878 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:45.275841    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:50.275719 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:50.275680    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:55:50.277326 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:50.277290    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" podUID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 20 10:55:56.580956 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:56.580889    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" containerID="cri-o://e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9" gracePeriod=45
Jul 20 10:55:56.760670 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:56.760620    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 20 10:55:57.326275 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.326247    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:55:57.429754 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.429660    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-home\") pod \"5b507d3d-e739-40f7-a447-d2759db56379\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") "
Jul 20 10:55:57.429754 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.429702    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-kserve-provision-location\") pod \"5b507d3d-e739-40f7-a447-d2759db56379\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") "
Jul 20 10:55:57.429754 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.429720    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-model-cache\") pod \"5b507d3d-e739-40f7-a447-d2759db56379\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") "
Jul 20 10:55:57.430045 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.429768    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-dshm\") pod \"5b507d3d-e739-40f7-a447-d2759db56379\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") "
Jul 20 10:55:57.430045 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.429794    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-tmp-dir\") pod \"5b507d3d-e739-40f7-a447-d2759db56379\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") "
Jul 20 10:55:57.430045 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.429832    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b507d3d-e739-40f7-a447-d2759db56379-tls-certs\") pod \"5b507d3d-e739-40f7-a447-d2759db56379\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") "
Jul 20 10:55:57.430045 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.429906    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzwpt\" (UniqueName: \"kubernetes.io/projected/5b507d3d-e739-40f7-a447-d2759db56379-kube-api-access-pzwpt\") pod \"5b507d3d-e739-40f7-a447-d2759db56379\" (UID: \"5b507d3d-e739-40f7-a447-d2759db56379\") "
Jul 20 10:55:57.430269 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.430149    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b507d3d-e739-40f7-a447-d2759db56379" (UID: "5b507d3d-e739-40f7-a447-d2759db56379"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:55:57.430432 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.430404    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-home" (OuterVolumeSpecName: "home") pod "5b507d3d-e739-40f7-a447-d2759db56379" (UID: "5b507d3d-e739-40f7-a447-d2759db56379"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:55:57.432140 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.432111    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b507d3d-e739-40f7-a447-d2759db56379-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b507d3d-e739-40f7-a447-d2759db56379" (UID: "5b507d3d-e739-40f7-a447-d2759db56379"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:55:57.432398 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.432365    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b507d3d-e739-40f7-a447-d2759db56379-kube-api-access-pzwpt" (OuterVolumeSpecName: "kube-api-access-pzwpt") pod "5b507d3d-e739-40f7-a447-d2759db56379" (UID: "5b507d3d-e739-40f7-a447-d2759db56379"). InnerVolumeSpecName "kube-api-access-pzwpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:55:57.432497 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.432477    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-dshm" (OuterVolumeSpecName: "dshm") pod "5b507d3d-e739-40f7-a447-d2759db56379" (UID: "5b507d3d-e739-40f7-a447-d2759db56379"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:55:57.448737 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.448705    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b507d3d-e739-40f7-a447-d2759db56379" (UID: "5b507d3d-e739-40f7-a447-d2759db56379"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:55:57.492311 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.492254    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b507d3d-e739-40f7-a447-d2759db56379" (UID: "5b507d3d-e739-40f7-a447-d2759db56379"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:55:57.531049 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.531022    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:55:57.531049 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.531046    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:55:57.531217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.531058    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:55:57.531217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.531067    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:55:57.531217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.531075    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b507d3d-e739-40f7-a447-d2759db56379-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:55:57.531217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.531083    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b507d3d-e739-40f7-a447-d2759db56379-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:55:57.531217 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.531091    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzwpt\" (UniqueName: \"kubernetes.io/projected/5b507d3d-e739-40f7-a447-d2759db56379-kube-api-access-pzwpt\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:55:57.630206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.630166    2572 generic.go:356] "Generic (PLEG): container finished" podID="5b507d3d-e739-40f7-a447-d2759db56379" containerID="e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9" exitCode=0
Jul 20 10:55:57.630636 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.630221    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"
Jul 20 10:55:57.630636 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.630241    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" event={"ID":"5b507d3d-e739-40f7-a447-d2759db56379","Type":"ContainerDied","Data":"e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9"}
Jul 20 10:55:57.630636 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.630296    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s" event={"ID":"5b507d3d-e739-40f7-a447-d2759db56379","Type":"ContainerDied","Data":"97cda0c7bf135e5ac3241545827b800f69cb59fcb8bae0144d69cfa602d4df60"}
Jul 20 10:55:57.630636 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.630318    2572 scope.go:117] "RemoveContainer" containerID="e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9"
Jul 20 10:55:57.639804 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.639784    2572 scope.go:117] "RemoveContainer" containerID="ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb"
Jul 20 10:55:57.648441 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.648420    2572 scope.go:117] "RemoveContainer" containerID="e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9"
Jul 20 10:55:57.648740 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:55:57.648720    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9\": container with ID starting with e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9 not found: ID does not exist" containerID="e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9"
Jul 20 10:55:57.648827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.648753    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9"} err="failed to get container status \"e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9\": rpc error: code = NotFound desc = could not find container \"e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9\": container with ID starting with e35d5259e18578a07c90457bb7a70c1a56638e26db499b2d57da99b59d58d4a9 not found: ID does not exist"
Jul 20 10:55:57.648827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.648777    2572 scope.go:117] "RemoveContainer" containerID="ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb"
Jul 20 10:55:57.649050 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:55:57.649032    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb\": container with ID starting with ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb not found: ID does not exist" containerID="ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb"
Jul 20 10:55:57.649091 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.649058    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb"} err="failed to get container status \"ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb\": rpc error: code = NotFound desc = could not find container \"ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb\": container with ID starting with ae338c6ee14024ebb72fb295b56c5b5926db39f218747bd3ca9f9d70c6796cfb not found: ID does not exist"
Jul 20 10:55:57.655445 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.655424    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"]
Jul 20 10:55:57.659448 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:57.659426    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-4668s"]
Jul 20 10:55:58.299964 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:58.299929    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"]
Jul 20 10:55:58.519115 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:55:58.519078    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b507d3d-e739-40f7-a447-d2759db56379" path="/var/lib/kubelet/pods/5b507d3d-e739-40f7-a447-d2759db56379/volumes"
Jul 20 10:56:13.323653 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:13.323604    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" podUID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerName="main" containerID="cri-o://5b424bebcde218ab37e9fcafb17d77e16d07bbd2a1bad755621e38ffb291e77b" gracePeriod=45
Jul 20 10:56:58.890400 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:58.890377    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6f4448865d-s9dcr_fc82f3c7-aec3-4d04-b1c5-1122a93e366f/main/0.log"
Jul 20 10:56:58.890899 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:58.890878    2572 generic.go:356] "Generic (PLEG): container finished" podID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerID="5b424bebcde218ab37e9fcafb17d77e16d07bbd2a1bad755621e38ffb291e77b" exitCode=137
Jul 20 10:56:58.890963 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:58.890946    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" event={"ID":"fc82f3c7-aec3-4d04-b1c5-1122a93e366f","Type":"ContainerDied","Data":"5b424bebcde218ab37e9fcafb17d77e16d07bbd2a1bad755621e38ffb291e77b"}
Jul 20 10:56:58.974466 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:58.974444    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6f4448865d-s9dcr_fc82f3c7-aec3-4d04-b1c5-1122a93e366f/main/0.log"
Jul 20 10:56:58.975904 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:58.975887    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:56:59.041895 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.041862    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-dshm\") pod \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") "
Jul 20 10:56:59.042047 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.041910    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn967\" (UniqueName: \"kubernetes.io/projected/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kube-api-access-bn967\") pod \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") "
Jul 20 10:56:59.042047 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.041948    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-model-cache\") pod \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") "
Jul 20 10:56:59.042047 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.041982    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tls-certs\") pod \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") "
Jul 20 10:56:59.042047 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.042002    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-home\") pod \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") "
Jul 20 10:56:59.042047 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.042018    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tmp-dir\") pod \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") "
Jul 20 10:56:59.042300 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.042126    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kserve-provision-location\") pod \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\" (UID: \"fc82f3c7-aec3-4d04-b1c5-1122a93e366f\") "
Jul 20 10:56:59.042300 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.042283    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc82f3c7-aec3-4d04-b1c5-1122a93e366f" (UID: "fc82f3c7-aec3-4d04-b1c5-1122a93e366f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:56:59.042410 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.042374    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc82f3c7-aec3-4d04-b1c5-1122a93e366f" (UID: "fc82f3c7-aec3-4d04-b1c5-1122a93e366f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:56:59.042410 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.042391    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:56:59.042531 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.042417    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-home" (OuterVolumeSpecName: "home") pod "fc82f3c7-aec3-4d04-b1c5-1122a93e366f" (UID: "fc82f3c7-aec3-4d04-b1c5-1122a93e366f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:56:59.044210 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.044186    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc82f3c7-aec3-4d04-b1c5-1122a93e366f" (UID: "fc82f3c7-aec3-4d04-b1c5-1122a93e366f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:56:59.044496 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.044476    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-dshm" (OuterVolumeSpecName: "dshm") pod "fc82f3c7-aec3-4d04-b1c5-1122a93e366f" (UID: "fc82f3c7-aec3-4d04-b1c5-1122a93e366f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:56:59.044496 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.044488    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kube-api-access-bn967" (OuterVolumeSpecName: "kube-api-access-bn967") pod "fc82f3c7-aec3-4d04-b1c5-1122a93e366f" (UID: "fc82f3c7-aec3-4d04-b1c5-1122a93e366f"). InnerVolumeSpecName "kube-api-access-bn967". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:56:59.099772 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.099689    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc82f3c7-aec3-4d04-b1c5-1122a93e366f" (UID: "fc82f3c7-aec3-4d04-b1c5-1122a93e366f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:56:59.143041 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.143009    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:56:59.143041 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.143037    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:56:59.143041 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.143045    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:56:59.143244 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.143053    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:56:59.143244 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.143063    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:56:59.143244 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.143072    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn967\" (UniqueName: \"kubernetes.io/projected/fc82f3c7-aec3-4d04-b1c5-1122a93e366f-kube-api-access-bn967\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:56:59.896436 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.896408    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6f4448865d-s9dcr_fc82f3c7-aec3-4d04-b1c5-1122a93e366f/main/0.log"
Jul 20 10:56:59.896864 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.896757    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr" event={"ID":"fc82f3c7-aec3-4d04-b1c5-1122a93e366f","Type":"ContainerDied","Data":"beb48ecf3206497e1f0e27724c4dcf91094a1c6dce34efa722f8037a2b94892d"}
Jul 20 10:56:59.896864 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.896789    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"
Jul 20 10:56:59.896864 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.896814    2572 scope.go:117] "RemoveContainer" containerID="5b424bebcde218ab37e9fcafb17d77e16d07bbd2a1bad755621e38ffb291e77b"
Jul 20 10:56:59.905466 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.905447    2572 scope.go:117] "RemoveContainer" containerID="9c005517e8bcdbcb469e52d14289a3499174093f2d848d409bbe1c0cc1689f2b"
Jul 20 10:56:59.919174 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.919150    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"]
Jul 20 10:56:59.922722 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:56:59.922701    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f4448865d-s9dcr"]
Jul 20 10:57:00.519875 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:00.519845    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" path="/var/lib/kubelet/pods/fc82f3c7-aec3-4d04-b1c5-1122a93e366f/volumes"
Jul 20 10:57:00.657843 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:00.657816    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:57:00.677249 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:00.677222    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 10:57:15.399255 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399221    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"]
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399504    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerName="main"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399514    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerName="main"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399526    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerName="storage-initializer"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399531    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerName="storage-initializer"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399544    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399549    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399573    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="storage-initializer"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399579    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="storage-initializer"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399628    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc82f3c7-aec3-4d04-b1c5-1122a93e366f" containerName="main"
Jul 20 10:57:15.399678 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.399636    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b507d3d-e739-40f7-a447-d2759db56379" containerName="main"
Jul 20 10:57:15.401601 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.401583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.404659 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.404637    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 20 10:57:15.406227 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.406207    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 20 10:57:15.406349 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.406207    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-v85r6\""
Jul 20 10:57:15.406349 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.406270    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 20 10:57:15.411100 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.411071    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"]
Jul 20 10:57:15.466992 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.466960    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.466992 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.466993    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.467233 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.467013    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbz4h\" (UniqueName: \"kubernetes.io/projected/1273e91c-d917-45ca-8bc1-e3f04b9f1918-kube-api-access-bbz4h\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.467233 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.467040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.467233 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.467083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.467233 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.467116    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.567940 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.567903    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.567940 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.567939    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.568206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.567960    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbz4h\" (UniqueName: \"kubernetes.io/projected/1273e91c-d917-45ca-8bc1-e3f04b9f1918-kube-api-access-bbz4h\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.568206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.568086    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.568206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.568137    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.568380 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.568214    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.568520 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.568492    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.568634 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.568593    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.568634 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.568602    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.570311 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.570281    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.570606 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.570588    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.576076 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.576056    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbz4h\" (UniqueName: \"kubernetes.io/projected/1273e91c-d917-45ca-8bc1-e3f04b9f1918-kube-api-access-bbz4h\") pod \"rolling-upgrade-test-kserve-74b79d7f77-jj7r8\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.636685 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.636648    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"]
Jul 20 10:57:15.639415 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.639395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.642395 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.642375    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-nlnsj\""
Jul 20 10:57:15.646666 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.646647    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"]
Jul 20 10:57:15.669652 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.669577    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.669652 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.669623    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwv4h\" (UniqueName: \"kubernetes.io/projected/ea91163f-2cf1-4470-a86a-3dbff67f3afd-kube-api-access-gwv4h\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.669841 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.669704    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.712734 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.712706    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:15.771112 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.771083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.771271 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.771220    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.771271 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.771242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwv4h\" (UniqueName: \"kubernetes.io/projected/ea91163f-2cf1-4470-a86a-3dbff67f3afd-kube-api-access-gwv4h\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.771602 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.771530    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.774373 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.774340    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.783829 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.782154    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwv4h\" (UniqueName: \"kubernetes.io/projected/ea91163f-2cf1-4470-a86a-3dbff67f3afd-kube-api-access-gwv4h\") pod \"rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.844538 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.844520    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"]
Jul 20 10:57:15.846894 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:57:15.846869    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1273e91c_d917_45ca_8bc1_e3f04b9f1918.slice/crio-7ff074fd57161554fb533f70e4bef7a1a5fd8025bc0c351c47d312f85378acf6 WatchSource:0}: Error finding container 7ff074fd57161554fb533f70e4bef7a1a5fd8025bc0c351c47d312f85378acf6: Status 404 returned error can't find the container with id 7ff074fd57161554fb533f70e4bef7a1a5fd8025bc0c351c47d312f85378acf6
Jul 20 10:57:15.951073 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.951039    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:15.958478 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.958453    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" event={"ID":"1273e91c-d917-45ca-8bc1-e3f04b9f1918","Type":"ContainerStarted","Data":"15e6ba3741645cad0df44a513b2db73a6dd012483db00752ee15cf18ce96712d"}
Jul 20 10:57:15.958676 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.958500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" event={"ID":"1273e91c-d917-45ca-8bc1-e3f04b9f1918","Type":"ContainerStarted","Data":"7ff074fd57161554fb533f70e4bef7a1a5fd8025bc0c351c47d312f85378acf6"}
Jul 20 10:57:15.975952 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:15.975912    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" podStartSLOduration=0.975898404 podStartE2EDuration="975.898404ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:57:15.974902881 +0000 UTC m=+3615.988004779" watchObservedRunningTime="2026-07-20 10:57:15.975898404 +0000 UTC m=+3615.989000313"
Jul 20 10:57:16.077931 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:16.077907    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"]
Jul 20 10:57:16.079858 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:57:16.079833    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea91163f_2cf1_4470_a86a_3dbff67f3afd.slice/crio-e61144f857b8359c27a5dd90d21021d3cfa7686a058369a144cbf8937cc2eb29 WatchSource:0}: Error finding container e61144f857b8359c27a5dd90d21021d3cfa7686a058369a144cbf8937cc2eb29: Status 404 returned error can't find the container with id e61144f857b8359c27a5dd90d21021d3cfa7686a058369a144cbf8937cc2eb29
Jul 20 10:57:16.713833 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:16.713797    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:16.964771 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:16.964677    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v" event={"ID":"ea91163f-2cf1-4470-a86a-3dbff67f3afd","Type":"ContainerStarted","Data":"cb2cf6d9cd90f96cece64496f42600f4704b226cd922fbbd5f1feaae358f0bae"}
Jul 20 10:57:16.964771 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:16.964737    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v" event={"ID":"ea91163f-2cf1-4470-a86a-3dbff67f3afd","Type":"ContainerStarted","Data":"e61144f857b8359c27a5dd90d21021d3cfa7686a058369a144cbf8937cc2eb29"}
Jul 20 10:57:16.964985 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:16.964885    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:16.983010 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:16.982966    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v" podStartSLOduration=1.9829515789999999 podStartE2EDuration="1.982951579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:57:16.981904367 +0000 UTC m=+3616.995006264" watchObservedRunningTime="2026-07-20 10:57:16.982951579 +0000 UTC m=+3616.996053475"
Jul 20 10:57:25.713903 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:25.713864    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:25.726395 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:25.726367    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:26.011171 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:26.011080    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:47.970202 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:47.970172    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:57:49.588020 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.587983    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"]
Jul 20 10:57:49.591355 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.591331    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.602943 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.602919    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"]
Jul 20 10:57:49.744608 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.744548    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2wd\" (UniqueName: \"kubernetes.io/projected/e9546d8d-6e56-4864-a42c-f2eb0428d54b-kube-api-access-zr2wd\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.744773 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.744631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.744773 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.744672    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.744773 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.744698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.744773 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.744719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.744773 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.744751    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.845625 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.845513    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.845625 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.845570    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.845625 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.845604    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.845935 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.845629    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2wd\" (UniqueName: \"kubernetes.io/projected/e9546d8d-6e56-4864-a42c-f2eb0428d54b-kube-api-access-zr2wd\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.845935 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.845669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.845935 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.845727    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.846093 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.846035    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.846093 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.846036    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.846093 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.846081    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.848062 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.848036    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.848232 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.848216    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.854898 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.854868    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2wd\" (UniqueName: \"kubernetes.io/projected/e9546d8d-6e56-4864-a42c-f2eb0428d54b-kube-api-access-zr2wd\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-rs4bk\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:49.902979 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:49.902944    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:50.037283 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:50.037249    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"]
Jul 20 10:57:50.040466 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:57:50.040438    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9546d8d_6e56_4864_a42c_f2eb0428d54b.slice/crio-7938a9f5fd1e0e8a018b7bf5c3bac89b224c2d1b7ab43668d95e2f7dc2f1acca WatchSource:0}: Error finding container 7938a9f5fd1e0e8a018b7bf5c3bac89b224c2d1b7ab43668d95e2f7dc2f1acca: Status 404 returned error can't find the container with id 7938a9f5fd1e0e8a018b7bf5c3bac89b224c2d1b7ab43668d95e2f7dc2f1acca
Jul 20 10:57:50.092065 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:50.092033    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk" event={"ID":"e9546d8d-6e56-4864-a42c-f2eb0428d54b","Type":"ContainerStarted","Data":"7938a9f5fd1e0e8a018b7bf5c3bac89b224c2d1b7ab43668d95e2f7dc2f1acca"}
Jul 20 10:57:51.097963 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:51.097922    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk" event={"ID":"e9546d8d-6e56-4864-a42c-f2eb0428d54b","Type":"ContainerStarted","Data":"63b889675c727e592f6d121cbdd42d6afd2c029ef067ac4b012489faf0015f9d"}
Jul 20 10:57:51.118650 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:51.118596    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk" podStartSLOduration=2.118580157 podStartE2EDuration="2.118580157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:57:51.117095214 +0000 UTC m=+3651.130197112" watchObservedRunningTime="2026-07-20 10:57:51.118580157 +0000 UTC m=+3651.131682043"
Jul 20 10:57:51.903898 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:51.903847    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:55.932108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:55.932071    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"]
Jul 20 10:57:55.938777 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:55.938748    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"]
Jul 20 10:57:55.941970 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:57:55.941938    2572 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:57:55.941970 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:57:55.941970 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:57:55.941970 ip-10-0-138-196 kubenswrapper[2572]:  > containerID="15e6ba3741645cad0df44a513b2db73a6dd012483db00752ee15cf18ce96712d" cmd=["/bin/sleep","15"]
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:55.941985    2572 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" message="" err=<
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]:  >
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:57:55.941996    2572 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" podUID="1273e91c-d917-45ca-8bc1-e3f04b9f1918" containerName="main" containerID="cri-o://15e6ba3741645cad0df44a513b2db73a6dd012483db00752ee15cf18ce96712d"
Jul 20 10:57:55.942266 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:55.942036    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" podUID="1273e91c-d917-45ca-8bc1-e3f04b9f1918" containerName="main" containerID="cri-o://15e6ba3741645cad0df44a513b2db73a6dd012483db00752ee15cf18ce96712d" gracePeriod=60
Jul 20 10:57:55.945430 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:55.945404    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"]
Jul 20 10:57:55.949246 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:57:55.949216    2572 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:57:55.949246 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:57:55.949246 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:57:55.949246 ip-10-0-138-196 kubenswrapper[2572]:  > containerID="63b889675c727e592f6d121cbdd42d6afd2c029ef067ac4b012489faf0015f9d" cmd=["/bin/sleep","15"]
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:55.949264    2572 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk" message="" err=<
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]:  >
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:57:55.949280    2572 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk" podUID="e9546d8d-6e56-4864-a42c-f2eb0428d54b" containerName="main" containerID="cri-o://63b889675c727e592f6d121cbdd42d6afd2c029ef067ac4b012489faf0015f9d"
Jul 20 10:57:55.949502 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:55.949318    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk" podUID="e9546d8d-6e56-4864-a42c-f2eb0428d54b" containerName="main" containerID="cri-o://63b889675c727e592f6d121cbdd42d6afd2c029ef067ac4b012489faf0015f9d" gracePeriod=60
Jul 20 10:57:56.000447 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.000406    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" podUID="1273e91c-d917-45ca-8bc1-e3f04b9f1918" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 20 10:57:56.119780 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.119745    2572 generic.go:356] "Generic (PLEG): container finished" podID="1273e91c-d917-45ca-8bc1-e3f04b9f1918" containerID="15e6ba3741645cad0df44a513b2db73a6dd012483db00752ee15cf18ce96712d" exitCode=0
Jul 20 10:57:56.119927 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.119812    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" event={"ID":"1273e91c-d917-45ca-8bc1-e3f04b9f1918","Type":"ContainerDied","Data":"15e6ba3741645cad0df44a513b2db73a6dd012483db00752ee15cf18ce96712d"}
Jul 20 10:57:56.121465 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.121441    2572 generic.go:356] "Generic (PLEG): container finished" podID="e9546d8d-6e56-4864-a42c-f2eb0428d54b" containerID="63b889675c727e592f6d121cbdd42d6afd2c029ef067ac4b012489faf0015f9d" exitCode=0
Jul 20 10:57:56.121628 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.121489    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk" event={"ID":"e9546d8d-6e56-4864-a42c-f2eb0428d54b","Type":"ContainerDied","Data":"63b889675c727e592f6d121cbdd42d6afd2c029ef067ac4b012489faf0015f9d"}
Jul 20 10:57:56.220474 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.220447    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:56.223357 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.223339    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:56.398628 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398593    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-home\") pod \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") "
Jul 20 10:57:56.398827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398649    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tmp-dir\") pod \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") "
Jul 20 10:57:56.398827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398677    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbz4h\" (UniqueName: \"kubernetes.io/projected/1273e91c-d917-45ca-8bc1-e3f04b9f1918-kube-api-access-bbz4h\") pod \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") "
Jul 20 10:57:56.398827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398708    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-dshm\") pod \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") "
Jul 20 10:57:56.398827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398759    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tls-certs\") pod \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") "
Jul 20 10:57:56.398827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398786    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr2wd\" (UniqueName: \"kubernetes.io/projected/e9546d8d-6e56-4864-a42c-f2eb0428d54b-kube-api-access-zr2wd\") pod \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") "
Jul 20 10:57:56.398827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398808    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tls-certs\") pod \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") "
Jul 20 10:57:56.399143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398830    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-home\") pod \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") "
Jul 20 10:57:56.399143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398868    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-model-cache\") pod \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") "
Jul 20 10:57:56.399143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398889    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-home" (OuterVolumeSpecName: "home") pod "1273e91c-d917-45ca-8bc1-e3f04b9f1918" (UID: "1273e91c-d917-45ca-8bc1-e3f04b9f1918"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:56.399143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398916    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-model-cache\") pod \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\" (UID: \"1273e91c-d917-45ca-8bc1-e3f04b9f1918\") "
Jul 20 10:57:56.399143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398922    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1273e91c-d917-45ca-8bc1-e3f04b9f1918" (UID: "1273e91c-d917-45ca-8bc1-e3f04b9f1918"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:56.399143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398942    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-dshm\") pod \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") "
Jul 20 10:57:56.399143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.398984    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tmp-dir\") pod \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\" (UID: \"e9546d8d-6e56-4864-a42c-f2eb0428d54b\") "
Jul 20 10:57:56.399143 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.399073    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-home" (OuterVolumeSpecName: "home") pod "e9546d8d-6e56-4864-a42c-f2eb0428d54b" (UID: "e9546d8d-6e56-4864-a42c-f2eb0428d54b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:56.399584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.399144    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9546d8d-6e56-4864-a42c-f2eb0428d54b" (UID: "e9546d8d-6e56-4864-a42c-f2eb0428d54b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:56.399584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.399269    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.399584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.399286    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.399584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.399290    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-model-cache" (OuterVolumeSpecName: "model-cache") pod "1273e91c-d917-45ca-8bc1-e3f04b9f1918" (UID: "1273e91c-d917-45ca-8bc1-e3f04b9f1918"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:56.399584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.399299    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.399584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.399321    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9546d8d-6e56-4864-a42c-f2eb0428d54b" (UID: "e9546d8d-6e56-4864-a42c-f2eb0428d54b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:56.399584 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.399333    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.401206 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.401183    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-dshm" (OuterVolumeSpecName: "dshm") pod "1273e91c-d917-45ca-8bc1-e3f04b9f1918" (UID: "1273e91c-d917-45ca-8bc1-e3f04b9f1918"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:56.401336 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.401279    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-dshm" (OuterVolumeSpecName: "dshm") pod "e9546d8d-6e56-4864-a42c-f2eb0428d54b" (UID: "e9546d8d-6e56-4864-a42c-f2eb0428d54b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:56.401505 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.401483    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9546d8d-6e56-4864-a42c-f2eb0428d54b-kube-api-access-zr2wd" (OuterVolumeSpecName: "kube-api-access-zr2wd") pod "e9546d8d-6e56-4864-a42c-f2eb0428d54b" (UID: "e9546d8d-6e56-4864-a42c-f2eb0428d54b"). InnerVolumeSpecName "kube-api-access-zr2wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:57:56.401505 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.401491    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9546d8d-6e56-4864-a42c-f2eb0428d54b" (UID: "e9546d8d-6e56-4864-a42c-f2eb0428d54b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:57:56.401629 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.401514    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1273e91c-d917-45ca-8bc1-e3f04b9f1918-kube-api-access-bbz4h" (OuterVolumeSpecName: "kube-api-access-bbz4h") pod "1273e91c-d917-45ca-8bc1-e3f04b9f1918" (UID: "1273e91c-d917-45ca-8bc1-e3f04b9f1918"). InnerVolumeSpecName "kube-api-access-bbz4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:57:56.401629 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.401601    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1273e91c-d917-45ca-8bc1-e3f04b9f1918" (UID: "1273e91c-d917-45ca-8bc1-e3f04b9f1918"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:57:56.499902 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.499869    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.499902 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.499900    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2wd\" (UniqueName: \"kubernetes.io/projected/e9546d8d-6e56-4864-a42c-f2eb0428d54b-kube-api-access-zr2wd\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.500108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.499911    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1273e91c-d917-45ca-8bc1-e3f04b9f1918-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.500108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.499920    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.500108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.499927    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.500108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.499934    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9546d8d-6e56-4864-a42c-f2eb0428d54b-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.500108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.499942    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbz4h\" (UniqueName: \"kubernetes.io/projected/1273e91c-d917-45ca-8bc1-e3f04b9f1918-kube-api-access-bbz4h\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:56.500108 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:56.499951    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1273e91c-d917-45ca-8bc1-e3f04b9f1918-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:57:57.127526 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.127495    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"
Jul 20 10:57:57.127526 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.127518    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk" event={"ID":"e9546d8d-6e56-4864-a42c-f2eb0428d54b","Type":"ContainerDied","Data":"7938a9f5fd1e0e8a018b7bf5c3bac89b224c2d1b7ab43668d95e2f7dc2f1acca"}
Jul 20 10:57:57.128024 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.127608    2572 scope.go:117] "RemoveContainer" containerID="63b889675c727e592f6d121cbdd42d6afd2c029ef067ac4b012489faf0015f9d"
Jul 20 10:57:57.128996 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.128891    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"
Jul 20 10:57:57.128996 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.128884    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8" event={"ID":"1273e91c-d917-45ca-8bc1-e3f04b9f1918","Type":"ContainerDied","Data":"7ff074fd57161554fb533f70e4bef7a1a5fd8025bc0c351c47d312f85378acf6"}
Jul 20 10:57:57.136804 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.136780    2572 scope.go:117] "RemoveContainer" containerID="15e6ba3741645cad0df44a513b2db73a6dd012483db00752ee15cf18ce96712d"
Jul 20 10:57:57.146497 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.146468    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"]
Jul 20 10:57:57.151512 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.151490    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-rs4bk"]
Jul 20 10:57:57.163142 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.163111    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"]
Jul 20 10:57:57.166024 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.166005    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-jj7r8"]
Jul 20 10:57:57.969472 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:57.969428    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v" podUID="ea91163f-2cf1-4470-a86a-3dbff67f3afd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:57:58.519045 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:58.519015    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1273e91c-d917-45ca-8bc1-e3f04b9f1918" path="/var/lib/kubelet/pods/1273e91c-d917-45ca-8bc1-e3f04b9f1918/volumes"
Jul 20 10:57:58.519405 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:57:58.519372    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9546d8d-6e56-4864-a42c-f2eb0428d54b" path="/var/lib/kubelet/pods/e9546d8d-6e56-4864-a42c-f2eb0428d54b/volumes"
Jul 20 10:58:07.969913 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:07.969850    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v" podUID="ea91163f-2cf1-4470-a86a-3dbff67f3afd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:58:10.966962 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:10.966918    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v" podUID="ea91163f-2cf1-4470-a86a-3dbff67f3afd" containerName="main" containerID="cri-o://cb2cf6d9cd90f96cece64496f42600f4704b226cd922fbbd5f1feaae358f0bae" gracePeriod=45
Jul 20 10:58:11.184433 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.184399    2572 generic.go:356] "Generic (PLEG): container finished" podID="ea91163f-2cf1-4470-a86a-3dbff67f3afd" containerID="cb2cf6d9cd90f96cece64496f42600f4704b226cd922fbbd5f1feaae358f0bae" exitCode=0
Jul 20 10:58:11.184614 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.184457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v" event={"ID":"ea91163f-2cf1-4470-a86a-3dbff67f3afd","Type":"ContainerDied","Data":"cb2cf6d9cd90f96cece64496f42600f4704b226cd922fbbd5f1feaae358f0bae"}
Jul 20 10:58:11.213078 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.213054    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:58:11.215002 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.214982    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tokenizer-uds\") pod \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") "
Jul 20 10:58:11.215089 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.215073    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwv4h\" (UniqueName: \"kubernetes.io/projected/ea91163f-2cf1-4470-a86a-3dbff67f3afd-kube-api-access-gwv4h\") pod \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") "
Jul 20 10:58:11.215139 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.215119    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tls-certs\") pod \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\" (UID: \"ea91163f-2cf1-4470-a86a-3dbff67f3afd\") "
Jul 20 10:58:11.215259 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.215241    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ea91163f-2cf1-4470-a86a-3dbff67f3afd" (UID: "ea91163f-2cf1-4470-a86a-3dbff67f3afd"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:58:11.215323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.215309    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:58:11.217080 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.217023    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea91163f-2cf1-4470-a86a-3dbff67f3afd-kube-api-access-gwv4h" (OuterVolumeSpecName: "kube-api-access-gwv4h") pod "ea91163f-2cf1-4470-a86a-3dbff67f3afd" (UID: "ea91163f-2cf1-4470-a86a-3dbff67f3afd"). InnerVolumeSpecName "kube-api-access-gwv4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:58:11.217173 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.217114    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea91163f-2cf1-4470-a86a-3dbff67f3afd" (UID: "ea91163f-2cf1-4470-a86a-3dbff67f3afd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:58:11.315686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.315650    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwv4h\" (UniqueName: \"kubernetes.io/projected/ea91163f-2cf1-4470-a86a-3dbff67f3afd-kube-api-access-gwv4h\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:58:11.315686 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:11.315680    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea91163f-2cf1-4470-a86a-3dbff67f3afd-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:58:12.189853 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:12.189821    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"
Jul 20 10:58:12.190333 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:12.189823    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v" event={"ID":"ea91163f-2cf1-4470-a86a-3dbff67f3afd","Type":"ContainerDied","Data":"e61144f857b8359c27a5dd90d21021d3cfa7686a058369a144cbf8937cc2eb29"}
Jul 20 10:58:12.190333 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:12.189972    2572 scope.go:117] "RemoveContainer" containerID="cb2cf6d9cd90f96cece64496f42600f4704b226cd922fbbd5f1feaae358f0bae"
Jul 20 10:58:12.212573 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:12.212530    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"]
Jul 20 10:58:12.216867 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:12.216837    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-777d56df88-vlf8v"]
Jul 20 10:58:12.519118 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:12.519085    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea91163f-2cf1-4470-a86a-3dbff67f3afd" path="/var/lib/kubelet/pods/ea91163f-2cf1-4470-a86a-3dbff67f3afd/volumes"
Jul 20 10:58:20.748336 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748296    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-687957b4c5-s2t9c"]
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748621    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea91163f-2cf1-4470-a86a-3dbff67f3afd" containerName="main"
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748635    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea91163f-2cf1-4470-a86a-3dbff67f3afd" containerName="main"
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748648    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1273e91c-d917-45ca-8bc1-e3f04b9f1918" containerName="main"
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748654    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1273e91c-d917-45ca-8bc1-e3f04b9f1918" containerName="main"
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748665    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9546d8d-6e56-4864-a42c-f2eb0428d54b" containerName="main"
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748674    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9546d8d-6e56-4864-a42c-f2eb0428d54b" containerName="main"
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748726    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea91163f-2cf1-4470-a86a-3dbff67f3afd" containerName="main"
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748733    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9546d8d-6e56-4864-a42c-f2eb0428d54b" containerName="main"
Jul 20 10:58:20.748842 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.748740    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1273e91c-d917-45ca-8bc1-e3f04b9f1918" containerName="main"
Jul 20 10:58:20.751233 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.751215    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:20.772929 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.772907    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-687957b4c5-s2t9c"]
Jul 20 10:58:20.778716 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.778689    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c826efa8-2802-4184-933f-f8a98556eed8-cert\") pod \"llmisvc-controller-manager-687957b4c5-s2t9c\" (UID: \"c826efa8-2802-4184-933f-f8a98556eed8\") " pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:20.778858 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.778732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5t79\" (UniqueName: \"kubernetes.io/projected/c826efa8-2802-4184-933f-f8a98556eed8-kube-api-access-b5t79\") pod \"llmisvc-controller-manager-687957b4c5-s2t9c\" (UID: \"c826efa8-2802-4184-933f-f8a98556eed8\") " pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:20.879265 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.879228    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c826efa8-2802-4184-933f-f8a98556eed8-cert\") pod \"llmisvc-controller-manager-687957b4c5-s2t9c\" (UID: \"c826efa8-2802-4184-933f-f8a98556eed8\") " pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:20.879265 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.879273    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5t79\" (UniqueName: \"kubernetes.io/projected/c826efa8-2802-4184-933f-f8a98556eed8-kube-api-access-b5t79\") pod \"llmisvc-controller-manager-687957b4c5-s2t9c\" (UID: \"c826efa8-2802-4184-933f-f8a98556eed8\") " pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:20.881734 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.881708    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c826efa8-2802-4184-933f-f8a98556eed8-cert\") pod \"llmisvc-controller-manager-687957b4c5-s2t9c\" (UID: \"c826efa8-2802-4184-933f-f8a98556eed8\") " pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:20.889830 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:20.889809    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5t79\" (UniqueName: \"kubernetes.io/projected/c826efa8-2802-4184-933f-f8a98556eed8-kube-api-access-b5t79\") pod \"llmisvc-controller-manager-687957b4c5-s2t9c\" (UID: \"c826efa8-2802-4184-933f-f8a98556eed8\") " pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:21.060749 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:21.060663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:21.189102 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:21.188950    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-687957b4c5-s2t9c"]
Jul 20 10:58:21.191775 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:58:21.191741    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc826efa8_2802_4184_933f_f8a98556eed8.slice/crio-7c556a966f9087e7cef3024d6b7b3e66bb996c755b4afb119bfe3aa45636f9fd WatchSource:0}: Error finding container 7c556a966f9087e7cef3024d6b7b3e66bb996c755b4afb119bfe3aa45636f9fd: Status 404 returned error can't find the container with id 7c556a966f9087e7cef3024d6b7b3e66bb996c755b4afb119bfe3aa45636f9fd
Jul 20 10:58:21.192987 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:21.192969    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:58:21.226161 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:21.226126    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c" event={"ID":"c826efa8-2802-4184-933f-f8a98556eed8","Type":"ContainerStarted","Data":"7c556a966f9087e7cef3024d6b7b3e66bb996c755b4afb119bfe3aa45636f9fd"}
Jul 20 10:58:22.231631 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:22.231597    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c" event={"ID":"c826efa8-2802-4184-933f-f8a98556eed8","Type":"ContainerStarted","Data":"1e84240d578eef2d0ab96ca4b80e23954239b0ac4934eb72ef3b78ceff577509"}
Jul 20 10:58:22.232129 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:22.231715    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:22.258878 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:22.258826    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c" podStartSLOduration=1.784719518 podStartE2EDuration="2.258808362s" totalImagesPullingTime="474.088844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:58:22.257481245 +0000 UTC m=+3682.270583142" watchObservedRunningTime="2026-07-20 10:58:22.258808362 +0000 UTC m=+3682.271910272"
Jul 20 10:58:53.236884 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.236852    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-687957b4c5-s2t9c"
Jul 20 10:58:53.290658 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.290626    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-5d54f9649b-7wc28"]
Jul 20 10:58:53.290849 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.290828    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28" podUID="145569bb-e0da-4065-886a-c3a8a5bcb6b4" containerName="manager" containerID="cri-o://9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f" gracePeriod=30
Jul 20 10:58:53.540664 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.540640    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:58:53.641973 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.641944    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145569bb-e0da-4065-886a-c3a8a5bcb6b4-cert\") pod \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\" (UID: \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\") "
Jul 20 10:58:53.642171 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.642029    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qrws\" (UniqueName: \"kubernetes.io/projected/145569bb-e0da-4065-886a-c3a8a5bcb6b4-kube-api-access-4qrws\") pod \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\" (UID: \"145569bb-e0da-4065-886a-c3a8a5bcb6b4\") "
Jul 20 10:58:53.644165 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.644135    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145569bb-e0da-4065-886a-c3a8a5bcb6b4-cert" (OuterVolumeSpecName: "cert") pod "145569bb-e0da-4065-886a-c3a8a5bcb6b4" (UID: "145569bb-e0da-4065-886a-c3a8a5bcb6b4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:58:53.644305 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.644229    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145569bb-e0da-4065-886a-c3a8a5bcb6b4-kube-api-access-4qrws" (OuterVolumeSpecName: "kube-api-access-4qrws") pod "145569bb-e0da-4065-886a-c3a8a5bcb6b4" (UID: "145569bb-e0da-4065-886a-c3a8a5bcb6b4"). InnerVolumeSpecName "kube-api-access-4qrws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:58:53.743076 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.743040    2572 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145569bb-e0da-4065-886a-c3a8a5bcb6b4-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:58:53.743076 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:53.743069    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrws\" (UniqueName: \"kubernetes.io/projected/145569bb-e0da-4065-886a-c3a8a5bcb6b4-kube-api-access-4qrws\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 10:58:54.359280 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.359245    2572 generic.go:356] "Generic (PLEG): container finished" podID="145569bb-e0da-4065-886a-c3a8a5bcb6b4" containerID="9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f" exitCode=0
Jul 20 10:58:54.359738 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.359291    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28"
Jul 20 10:58:54.359738 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.359332    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28" event={"ID":"145569bb-e0da-4065-886a-c3a8a5bcb6b4","Type":"ContainerDied","Data":"9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f"}
Jul 20 10:58:54.359738 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.359384    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5d54f9649b-7wc28" event={"ID":"145569bb-e0da-4065-886a-c3a8a5bcb6b4","Type":"ContainerDied","Data":"e4311ed3785e958960eab680f932d6e43ede7b3f62b806a949bb4636fde121b6"}
Jul 20 10:58:54.359738 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.359406    2572 scope.go:117] "RemoveContainer" containerID="9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f"
Jul 20 10:58:54.369431 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.369414    2572 scope.go:117] "RemoveContainer" containerID="9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f"
Jul 20 10:58:54.369668 ip-10-0-138-196 kubenswrapper[2572]: E0720 10:58:54.369646    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f\": container with ID starting with 9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f not found: ID does not exist" containerID="9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f"
Jul 20 10:58:54.369720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.369677    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f"} err="failed to get container status \"9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f\": rpc error: code = NotFound desc = could not find container \"9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f\": container with ID starting with 9b3bb2ca8c7efa470e276f9a2097d6a533415c12aa16a5bd99488f9dc61b828f not found: ID does not exist"
Jul 20 10:58:54.382812 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.382791    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-5d54f9649b-7wc28"]
Jul 20 10:58:54.386261 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.386240    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-5d54f9649b-7wc28"]
Jul 20 10:58:54.519118 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:58:54.519090    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145569bb-e0da-4065-886a-c3a8a5bcb6b4" path="/var/lib/kubelet/pods/145569bb-e0da-4065-886a-c3a8a5bcb6b4/volumes"
Jul 20 10:59:24.771691 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.771656    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"]
Jul 20 10:59:24.772100 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.771962    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145569bb-e0da-4065-886a-c3a8a5bcb6b4" containerName="manager"
Jul 20 10:59:24.772100 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.771979    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="145569bb-e0da-4065-886a-c3a8a5bcb6b4" containerName="manager"
Jul 20 10:59:24.772100 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.772073    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="145569bb-e0da-4065-886a-c3a8a5bcb6b4" containerName="manager"
Jul 20 10:59:24.774985 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.774967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.779543 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.779522    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Jul 20 10:59:24.780484 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.780463    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Jul 20 10:59:24.780665 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.780486    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-b8wjq\""
Jul 20 10:59:24.780665 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.780489    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Jul 20 10:59:24.792055 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792035    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-certs\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.792165 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792068    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-data\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.792165 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792090    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-envoy\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.792165 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792109    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-podinfo\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.792286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792177    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-token\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.792286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-socket\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.792286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.792286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792249    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t27z\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-kube-api-access-9t27z\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.792286 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.792274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-credential-socket\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.878786 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.878757    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"]
Jul 20 10:59:24.893099 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-credential-socket\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893242 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-certs\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893242 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893140    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-data\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893242 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893167    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-envoy\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893242 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-podinfo\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893242 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893216    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-token\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-socket\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893516 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893392    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t27z\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-kube-api-access-9t27z\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893575    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-data\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893642    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-certs\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893712 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893691    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-credential-socket\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.893849 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.893740    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-socket\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.894048 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.894031    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.895427 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.895407    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-envoy\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.895598 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.895580    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-podinfo\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.907323 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.907297    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-token\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:24.907541 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:24.907516    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t27z\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-kube-api-access-9t27z\") pod \"router-gateway-1-openshift-default-74fb5bd55-t7pzk\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:25.085474 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:25.085378    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:25.231226 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:25.231187    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"]
Jul 20 10:59:25.234935 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:59:25.234902    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bf7896_c6f7_476b_a70c_0a9711d67aaf.slice/crio-580b91604c327e781f6c21fe1663b48e05da715723c98b95795c424ff86bb01b WatchSource:0}: Error finding container 580b91604c327e781f6c21fe1663b48e05da715723c98b95795c424ff86bb01b: Status 404 returned error can't find the container with id 580b91604c327e781f6c21fe1663b48e05da715723c98b95795c424ff86bb01b
Jul 20 10:59:25.479919 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:25.479881    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" event={"ID":"a1bf7896-c6f7-476b-a70c-0a9711d67aaf","Type":"ContainerStarted","Data":"580b91604c327e781f6c21fe1663b48e05da715723c98b95795c424ff86bb01b"}
Jul 20 10:59:27.757776 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:27.757731    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 10:59:27.758078 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:27.757810    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 10:59:27.758078 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:27.757846    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 10:59:28.493703 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:28.493659    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" event={"ID":"a1bf7896-c6f7-476b-a70c-0a9711d67aaf","Type":"ContainerStarted","Data":"5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8"}
Jul 20 10:59:28.519487 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:28.519436    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podStartSLOduration=1.998751461 podStartE2EDuration="4.519418171s" totalImagesPullingTime="2.52066671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:59:28.518483292 +0000 UTC m=+3748.531585270" watchObservedRunningTime="2026-07-20 10:59:28.519418171 +0000 UTC m=+3748.532520071"
Jul 20 10:59:29.086589 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:29.086531    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:30.128753 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:30.128707    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:59:31.130827 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:31.130778    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:59:32.132285 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:32.132232    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:59:33.133526 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:33.133478    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:59:34.175796 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:34.175692    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:59:35.216748 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:35.216703    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:59:36.257773 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:36.257726    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:59:36.762604 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:36.762538    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": read tcp 10.134.0.2:58384->10.134.0.72:15021: read: connection reset by peer"
Jul 20 10:59:37.086044 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:37.085955    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 20 10:59:37.532175 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:37.532137    2572 generic.go:356] "Generic (PLEG): container finished" podID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerID="5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8" exitCode=0
Jul 20 10:59:37.532175 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:37.532178    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" event={"ID":"a1bf7896-c6f7-476b-a70c-0a9711d67aaf","Type":"ContainerDied","Data":"5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8"}
Jul 20 10:59:37.532727 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:37.532632    2572 scope.go:117] "RemoveContainer" containerID="5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8"
Jul 20 10:59:37.534474 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:37.534439    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 10:59:37.534572 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:37.534531    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 10:59:37.534621 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:37.534592    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 10:59:38.085806 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:38.085764    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:38.550136 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:38.550057    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" event={"ID":"a1bf7896-c6f7-476b-a70c-0a9711d67aaf","Type":"ContainerStarted","Data":"f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0"}
Jul 20 10:59:39.085854 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:39.085812    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:39.086157 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:39.086130    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 20 10:59:40.086501 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:40.086453    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:40.087031 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:40.086839    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 20 10:59:41.086431 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:41.086389    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 20 10:59:42.086319 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:42.086275    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 20 10:59:43.086726 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:43.086680    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 20 10:59:44.089072 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:44.089041    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:44.090050 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:44.090031    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 10:59:53.185446 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.185360    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"]
Jul 20 10:59:53.188612 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.188592    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.191252 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.191227    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Jul 20 10:59:53.191411 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.191383    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-rgl6d\""
Jul 20 10:59:53.191566 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.191484    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 20 10:59:53.200250 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.200226    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"]
Jul 20 10:59:53.358160 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.358119    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-model-cache\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.358160 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.358164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-dshm\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.358383 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.358198    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-home\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.358383 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.358243    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-tmp-dir\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.358383 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.358276    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-kserve-provision-location\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.358383 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.358301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/a6d84573-ce46-43ae-8cd2-79938a6de91b-kube-api-access-z8jxj\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.358522 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.358387    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d84573-ce46-43ae-8cd2-79938a6de91b-tls-certs\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459161 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459127    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-tmp-dir\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459330 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459170    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-kserve-provision-location\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459330 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459190    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/a6d84573-ce46-43ae-8cd2-79938a6de91b-kube-api-access-z8jxj\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459330 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459228    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d84573-ce46-43ae-8cd2-79938a6de91b-tls-certs\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459330 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-model-cache\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459330 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459271    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-dshm\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459330 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-home\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459667 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459647    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-tmp-dir\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459723 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459663    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-kserve-provision-location\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459765 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459729    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-home\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.459765 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.459740    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-model-cache\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.461701 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.461673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-dshm\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.461915 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.461899    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d84573-ce46-43ae-8cd2-79938a6de91b-tls-certs\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.469348 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.469327    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/a6d84573-ce46-43ae-8cd2-79938a6de91b-kube-api-access-z8jxj\") pod \"router-with-refs-test-kserve-669b4995d8-dkwd5\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.499314 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.499283    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 10:59:53.636720 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:53.636523    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"]
Jul 20 10:59:53.638885 ip-10-0-138-196 kubenswrapper[2572]: W0720 10:59:53.638854    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d84573_ce46_43ae_8cd2_79938a6de91b.slice/crio-0e45fe38c455bd3abffa3564e72a1dd4c148d838fc0f109633a52c1f6d9992b9 WatchSource:0}: Error finding container 0e45fe38c455bd3abffa3564e72a1dd4c148d838fc0f109633a52c1f6d9992b9: Status 404 returned error can't find the container with id 0e45fe38c455bd3abffa3564e72a1dd4c148d838fc0f109633a52c1f6d9992b9
Jul 20 10:59:54.616818 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:54.616783    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" event={"ID":"a6d84573-ce46-43ae-8cd2-79938a6de91b","Type":"ContainerStarted","Data":"461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653"}
Jul 20 10:59:54.617302 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:54.616843    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" event={"ID":"a6d84573-ce46-43ae-8cd2-79938a6de91b","Type":"ContainerStarted","Data":"0e45fe38c455bd3abffa3564e72a1dd4c148d838fc0f109633a52c1f6d9992b9"}
Jul 20 10:59:57.630139 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:57.630101    2572 generic.go:356] "Generic (PLEG): container finished" podID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerID="461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653" exitCode=0
Jul 20 10:59:57.630578 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:57.630170    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" event={"ID":"a6d84573-ce46-43ae-8cd2-79938a6de91b","Type":"ContainerDied","Data":"461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653"}
Jul 20 10:59:58.636733 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:58.636695    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" event={"ID":"a6d84573-ce46-43ae-8cd2-79938a6de91b","Type":"ContainerStarted","Data":"26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7"}
Jul 20 10:59:58.659127 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:58.659076    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podStartSLOduration=2.717943535 podStartE2EDuration="5.65906109s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.941117555s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:59:58.658225831 +0000 UTC m=+3778.671327728" watchObservedRunningTime="2026-07-20 10:59:58.65906109 +0000 UTC m=+3778.672162986"
Jul 20 10:59:59.500137 ip-10-0-138-196 kubenswrapper[2572]: I0720 10:59:59.500104    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 11:00:03.500059 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:00:03.500018    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 11:00:03.501819 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:00:03.501784    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:00:13.500451 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:00:13.500402    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:00:23.500223 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:00:23.500178    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:00:33.499859 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:00:33.499813    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:00:43.500748 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:00:43.500693    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:00:53.500299 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:00:53.500242    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:01:03.500012 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:01:03.499965    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:01:13.500462 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:01:13.500415    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:01:23.499824 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:01:23.499730    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:01:33.500624 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:01:33.500550    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:01:43.499939 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:01:43.499878    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:01:53.509952 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:01:53.509915    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 11:01:53.517899 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:01:53.517874    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 11:02:00.683905 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:02:00.683863    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:02:00.702210 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:02:00.702186    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:04:02.466130 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:02.466094    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"]
Jul 20 11:04:17.488981 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:17.488917    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" containerID="cri-o://26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7" gracePeriod=45
Jul 20 11:04:17.510777 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:17.510736    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 20 11:04:18.254606 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.254579    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 11:04:18.421537 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.421440    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-tmp-dir\") pod \"a6d84573-ce46-43ae-8cd2-79938a6de91b\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") "
Jul 20 11:04:18.421537 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.421486    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-kserve-provision-location\") pod \"a6d84573-ce46-43ae-8cd2-79938a6de91b\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") "
Jul 20 11:04:18.421537 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.421532    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/a6d84573-ce46-43ae-8cd2-79938a6de91b-kube-api-access-z8jxj\") pod \"a6d84573-ce46-43ae-8cd2-79938a6de91b\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") "
Jul 20 11:04:18.421851 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.421583    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-dshm\") pod \"a6d84573-ce46-43ae-8cd2-79938a6de91b\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") "
Jul 20 11:04:18.421851 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.421630    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-model-cache\") pod \"a6d84573-ce46-43ae-8cd2-79938a6de91b\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") "
Jul 20 11:04:18.421851 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.421655    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-home\") pod \"a6d84573-ce46-43ae-8cd2-79938a6de91b\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") "
Jul 20 11:04:18.421851 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.421674    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d84573-ce46-43ae-8cd2-79938a6de91b-tls-certs\") pod \"a6d84573-ce46-43ae-8cd2-79938a6de91b\" (UID: \"a6d84573-ce46-43ae-8cd2-79938a6de91b\") "
Jul 20 11:04:18.422072 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.421903    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6d84573-ce46-43ae-8cd2-79938a6de91b" (UID: "a6d84573-ce46-43ae-8cd2-79938a6de91b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:18.422320 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.422293    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-home" (OuterVolumeSpecName: "home") pod "a6d84573-ce46-43ae-8cd2-79938a6de91b" (UID: "a6d84573-ce46-43ae-8cd2-79938a6de91b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:18.424062 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.424027    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-dshm" (OuterVolumeSpecName: "dshm") pod "a6d84573-ce46-43ae-8cd2-79938a6de91b" (UID: "a6d84573-ce46-43ae-8cd2-79938a6de91b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:18.424184 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.424080    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d84573-ce46-43ae-8cd2-79938a6de91b-kube-api-access-z8jxj" (OuterVolumeSpecName: "kube-api-access-z8jxj") pod "a6d84573-ce46-43ae-8cd2-79938a6de91b" (UID: "a6d84573-ce46-43ae-8cd2-79938a6de91b"). InnerVolumeSpecName "kube-api-access-z8jxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:04:18.424264 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.424241    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d84573-ce46-43ae-8cd2-79938a6de91b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6d84573-ce46-43ae-8cd2-79938a6de91b" (UID: "a6d84573-ce46-43ae-8cd2-79938a6de91b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:04:18.434518 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.434487    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6d84573-ce46-43ae-8cd2-79938a6de91b" (UID: "a6d84573-ce46-43ae-8cd2-79938a6de91b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:18.484610 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.484545    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6d84573-ce46-43ae-8cd2-79938a6de91b" (UID: "a6d84573-ce46-43ae-8cd2-79938a6de91b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:18.522448 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.522414    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.522448 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.522441    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.522448 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.522449    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d84573-ce46-43ae-8cd2-79938a6de91b-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.522448 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.522457    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.522955 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.522465    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.522955 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.522475    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jxj\" (UniqueName: \"kubernetes.io/projected/a6d84573-ce46-43ae-8cd2-79938a6de91b-kube-api-access-z8jxj\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.522955 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.522483    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d84573-ce46-43ae-8cd2-79938a6de91b-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:18.694208 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.694112    2572 generic.go:356] "Generic (PLEG): container finished" podID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerID="26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7" exitCode=0
Jul 20 11:04:18.694208 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.694173    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"
Jul 20 11:04:18.694208 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.694181    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" event={"ID":"a6d84573-ce46-43ae-8cd2-79938a6de91b","Type":"ContainerDied","Data":"26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7"}
Jul 20 11:04:18.694506 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.694225    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5" event={"ID":"a6d84573-ce46-43ae-8cd2-79938a6de91b","Type":"ContainerDied","Data":"0e45fe38c455bd3abffa3564e72a1dd4c148d838fc0f109633a52c1f6d9992b9"}
Jul 20 11:04:18.694506 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.694246    2572 scope.go:117] "RemoveContainer" containerID="26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7"
Jul 20 11:04:18.702617 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.702604    2572 scope.go:117] "RemoveContainer" containerID="461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653"
Jul 20 11:04:18.710830 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.710808    2572 scope.go:117] "RemoveContainer" containerID="26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7"
Jul 20 11:04:18.711148 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:04:18.711116    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7\": container with ID starting with 26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7 not found: ID does not exist" containerID="26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7"
Jul 20 11:04:18.711266 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.711153    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7"} err="failed to get container status \"26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7\": rpc error: code = NotFound desc = could not find container \"26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7\": container with ID starting with 26125cea2c93867c9c609c4608ca8ff4c8ab76296f09fdfe77ec28c85f113fb7 not found: ID does not exist"
Jul 20 11:04:18.711266 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.711177    2572 scope.go:117] "RemoveContainer" containerID="461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653"
Jul 20 11:04:18.711491 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:04:18.711470    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653\": container with ID starting with 461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653 not found: ID does not exist" containerID="461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653"
Jul 20 11:04:18.711571 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.711500    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653"} err="failed to get container status \"461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653\": rpc error: code = NotFound desc = could not find container \"461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653\": container with ID starting with 461f8e53aed17d67413d687e131bb6fa42e9066fd7ea5e0d82bdd4bf66067653 not found: ID does not exist"
Jul 20 11:04:18.712579 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.712541    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"]
Jul 20 11:04:18.715788 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:18.715765    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-669b4995d8-dkwd5"]
Jul 20 11:04:20.519634 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:20.519600    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" path="/var/lib/kubelet/pods/a6d84573-ce46-43ae-8cd2-79938a6de91b/volumes"
Jul 20 11:04:27.416921 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:27.416885    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"]
Jul 20 11:04:27.417396 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:27.417084    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" containerID="cri-o://f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0" gracePeriod=30
Jul 20 11:04:29.091638 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:29.091588    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 11:04:32.669887 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.669857    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 11:04:32.724474 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724439    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-token\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.724680 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724482    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-podinfo\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.724680 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724506    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-certs\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.724680 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724544    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istiod-ca-cert\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.724680 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724585    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-credential-socket\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.724680 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724617    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t27z\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-kube-api-access-9t27z\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.724680 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724644    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-socket\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.724680 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724669    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-data\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.725079 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724722    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-envoy\") pod \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\" (UID: \"a1bf7896-c6f7-476b-a70c-0a9711d67aaf\") "
Jul 20 11:04:32.725079 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724856    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:32.725079 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.724971    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:32.725079 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.725009    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:32.725079 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.725044    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:32.725396 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.725078    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:04:32.725396 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.725363    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:32.727157 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.727135    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:04:32.727343 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.727307    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:32.727343 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.727317    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-kube-api-access-9t27z" (OuterVolumeSpecName: "kube-api-access-9t27z") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "kube-api-access-9t27z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:04:32.727657 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.727637    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1bf7896-c6f7-476b-a70c-0a9711d67aaf" (UID: "a1bf7896-c6f7-476b-a70c-0a9711d67aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 11:04:32.748662 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.748624    2572 generic.go:356] "Generic (PLEG): container finished" podID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerID="f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0" exitCode=0
Jul 20 11:04:32.748826 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.748672    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"
Jul 20 11:04:32.748826 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.748696    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" event={"ID":"a1bf7896-c6f7-476b-a70c-0a9711d67aaf","Type":"ContainerDied","Data":"f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0"}
Jul 20 11:04:32.748826 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.748748    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk" event={"ID":"a1bf7896-c6f7-476b-a70c-0a9711d67aaf","Type":"ContainerDied","Data":"580b91604c327e781f6c21fe1663b48e05da715723c98b95795c424ff86bb01b"}
Jul 20 11:04:32.748826 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.748769    2572 scope.go:117] "RemoveContainer" containerID="f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0"
Jul 20 11:04:32.758342 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.758325    2572 scope.go:117] "RemoveContainer" containerID="5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8"
Jul 20 11:04:32.765932 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.765914    2572 scope.go:117] "RemoveContainer" containerID="f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0"
Jul 20 11:04:32.766199 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:04:32.766179    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0\": container with ID starting with f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0 not found: ID does not exist" containerID="f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0"
Jul 20 11:04:32.766263 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.766206    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0"} err="failed to get container status \"f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0\": rpc error: code = NotFound desc = could not find container \"f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0\": container with ID starting with f1edf2a126a0ff541a94bee246c8751d746b36bcd0ad45a506e6ecd24661abc0 not found: ID does not exist"
Jul 20 11:04:32.766263 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.766226    2572 scope.go:117] "RemoveContainer" containerID="5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8"
Jul 20 11:04:32.766465 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:04:32.766449    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8\": container with ID starting with 5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8 not found: ID does not exist" containerID="5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8"
Jul 20 11:04:32.766506 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.766469    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8"} err="failed to get container status \"5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8\": rpc error: code = NotFound desc = could not find container \"5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8\": container with ID starting with 5bc50279a202851bd920a92b8fca646da9080b11a7f3678a9093e5b0ac50eee8 not found: ID does not exist"
Jul 20 11:04:32.770228 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.770207    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"]
Jul 20 11:04:32.776178 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.776154    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-74fb5bd55-t7pzk"]
Jul 20 11:04:32.826227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.826172    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istiod-ca-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:32.826227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.826219    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-credential-socket\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:32.826227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.826232    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t27z\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-kube-api-access-9t27z\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:32.826227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.826244    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-workload-socket\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:32.826515 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.826254    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-data\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:32.826515 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.826263    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-envoy\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:32.826515 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.826270    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-token\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:32.826515 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:32.826277    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf7896-c6f7-476b-a70c-0a9711d67aaf-istio-podinfo\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:04:34.519935 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:34.519900    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" path="/var/lib/kubelet/pods/a1bf7896-c6f7-476b-a70c-0a9711d67aaf/volumes"
Jul 20 11:04:37.427751 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.427713    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"]
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428011    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="storage-initializer"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428023    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="storage-initializer"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428041    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428047    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428054    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428060    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428077    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428085    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428133    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6d84573-ce46-43ae-8cd2-79938a6de91b" containerName="main"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428141    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy"
Jul 20 11:04:37.429919 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.428154    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1bf7896-c6f7-476b-a70c-0a9711d67aaf" containerName="istio-proxy"
Jul 20 11:04:37.431083 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.431062    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.434008 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.433968    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Jul 20 11:04:37.434165 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.434111    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-w8gst\""
Jul 20 11:04:37.434417 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.434398    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-4vjfx\""
Jul 20 11:04:37.434512 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.434404    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Jul 20 11:04:37.435196 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.435178    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Jul 20 11:04:37.435295 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.435256    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 20 11:04:37.448260 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.448234    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"]
Jul 20 11:04:37.454269 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.454249    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"]
Jul 20 11:04:37.456382 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.456362    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.456488 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.456398    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.456488 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.456463    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.456628 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.456502    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.456628 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.456528    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj2rf\" (UniqueName: \"kubernetes.io/projected/bf97e47c-4717-4ed3-b2de-16b83f087651-kube-api-access-qj2rf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.456628 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.456587    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf97e47c-4717-4ed3-b2de-16b83f087651-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.456740 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.456676    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.457178 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.457164    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.471643 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.471615    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"]
Jul 20 11:04:37.557451 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.557411    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf97e47c-4717-4ed3-b2de-16b83f087651-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.557687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.557477    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.557687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.557501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.557687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.557518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.557687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.557639    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.557687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.557680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.557992 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.557699    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj2rf\" (UniqueName: \"kubernetes.io/projected/bf97e47c-4717-4ed3-b2de-16b83f087651-kube-api-access-qj2rf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.557992 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.557893    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.558120 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.558041    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.558120 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.558086    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.558120 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.558108    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.560040 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.560019    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.560330 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.560308    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf97e47c-4717-4ed3-b2de-16b83f087651-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.567992 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.567965    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj2rf\" (UniqueName: \"kubernetes.io/projected/bf97e47c-4717-4ed3-b2de-16b83f087651-kube-api-access-qj2rf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.658619 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.658547    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.658810 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.658667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chfpp\" (UniqueName: \"kubernetes.io/projected/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kube-api-access-chfpp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.658810 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.658705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.658810 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.658732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.658810 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.658749    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.658810 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.658773    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.659038 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.658828    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.741259 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.741214    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:37.759439 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.759404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.759587 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.759457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.759587 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.759486    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.759587 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.759545    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.759770 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.759621    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chfpp\" (UniqueName: \"kubernetes.io/projected/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kube-api-access-chfpp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.759890 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.759836    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.759890 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.759870    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.760080 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.759957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.760080 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.760007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.760238 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.760213    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.760349 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.760280    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.762371 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.762349    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.762640 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.762622    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.769606 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.769583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chfpp\" (UniqueName: \"kubernetes.io/projected/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kube-api-access-chfpp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:37.881079 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:37.881043    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"]
Jul 20 11:04:37.884007 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:04:37.883978    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf97e47c_4717_4ed3_b2de_16b83f087651.slice/crio-9cc33d74e59784057a8b114d0cd0a653a0a9a9492e1d8f9b0a9ca67bf08d0f5a WatchSource:0}: Error finding container 9cc33d74e59784057a8b114d0cd0a653a0a9a9492e1d8f9b0a9ca67bf08d0f5a: Status 404 returned error can't find the container with id 9cc33d74e59784057a8b114d0cd0a653a0a9a9492e1d8f9b0a9ca67bf08d0f5a
Jul 20 11:04:38.067652 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:38.067545    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:38.212357 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:38.212331    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"]
Jul 20 11:04:38.214524 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:04:38.214498    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43a41cb_5732_4fb9_aba6_9f8c9267f74f.slice/crio-67e527919177ad3ccaa55f29f7070ba41038c824b7addce377064fadfe096d89 WatchSource:0}: Error finding container 67e527919177ad3ccaa55f29f7070ba41038c824b7addce377064fadfe096d89: Status 404 returned error can't find the container with id 67e527919177ad3ccaa55f29f7070ba41038c824b7addce377064fadfe096d89
Jul 20 11:04:38.776277 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:38.776238    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" event={"ID":"bf97e47c-4717-4ed3-b2de-16b83f087651","Type":"ContainerStarted","Data":"72fb2d7a1641c8e5450c259b9c920026eb9b9e114190139058f8df0abd58b2b6"}
Jul 20 11:04:38.776737 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:38.776297    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" event={"ID":"bf97e47c-4717-4ed3-b2de-16b83f087651","Type":"ContainerStarted","Data":"9cc33d74e59784057a8b114d0cd0a653a0a9a9492e1d8f9b0a9ca67bf08d0f5a"}
Jul 20 11:04:38.776737 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:38.776325    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:38.777908 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:38.777882    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" event={"ID":"a43a41cb-5732-4fb9-aba6-9f8c9267f74f","Type":"ContainerStarted","Data":"bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0"}
Jul 20 11:04:38.778022 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:38.777929    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" event={"ID":"a43a41cb-5732-4fb9-aba6-9f8c9267f74f","Type":"ContainerStarted","Data":"67e527919177ad3ccaa55f29f7070ba41038c824b7addce377064fadfe096d89"}
Jul 20 11:04:39.785978 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:39.785923    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" event={"ID":"bf97e47c-4717-4ed3-b2de-16b83f087651","Type":"ContainerStarted","Data":"3458300fe85a09b1c3ff781db656665195635cc4a56c70ee38a011cbd1347d00"}
Jul 20 11:04:41.801526 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:41.801491    2572 generic.go:356] "Generic (PLEG): container finished" podID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerID="bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0" exitCode=0
Jul 20 11:04:41.801928 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:41.801538    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" event={"ID":"a43a41cb-5732-4fb9-aba6-9f8c9267f74f","Type":"ContainerDied","Data":"bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0"}
Jul 20 11:04:42.808102 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:42.808054    2572 generic.go:356] "Generic (PLEG): container finished" podID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerID="3458300fe85a09b1c3ff781db656665195635cc4a56c70ee38a011cbd1347d00" exitCode=0
Jul 20 11:04:42.808541 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:42.808129    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" event={"ID":"bf97e47c-4717-4ed3-b2de-16b83f087651","Type":"ContainerDied","Data":"3458300fe85a09b1c3ff781db656665195635cc4a56c70ee38a011cbd1347d00"}
Jul 20 11:04:42.810171 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:42.810145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" event={"ID":"a43a41cb-5732-4fb9-aba6-9f8c9267f74f","Type":"ContainerStarted","Data":"a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78"}
Jul 20 11:04:42.860052 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:42.859982    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podStartSLOduration=2.878699481 podStartE2EDuration="5.859959658s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.981260177s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:04:42.857905506 +0000 UTC m=+4062.871007403" watchObservedRunningTime="2026-07-20 11:04:42.859959658 +0000 UTC m=+4062.873061555"
Jul 20 11:04:43.817621 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:43.817585    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" event={"ID":"bf97e47c-4717-4ed3-b2de-16b83f087651","Type":"ContainerStarted","Data":"acf8c15c095f580ede1eb3045e0b83e1be4a168d3e7359a81ebbd9cf05e5259b"}
Jul 20 11:04:43.843210 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:43.843141    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podStartSLOduration=3.542513482 podStartE2EDuration="6.843119258s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.300605776s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:04:43.84137351 +0000 UTC m=+4063.854475425" watchObservedRunningTime="2026-07-20 11:04:43.843119258 +0000 UTC m=+4063.856221155"
Jul 20 11:04:47.741502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:47.741452    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:47.741983 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:47.741531    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:47.742945 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:47.742907    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:04:47.754497 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:47.754470    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:04:48.068638 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:48.068524    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:48.068638 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:48.068612    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:04:48.070264 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:48.070229    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:04:57.742512 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:57.742449    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:04:58.068983 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:04:58.068880    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:05:07.742791 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:07.742737    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:05:08.068697 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:08.068579    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:05:17.742350 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:17.742290    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:05:18.068792 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:18.068690    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:05:27.742383 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:27.742336    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:05:28.068351 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:28.068238    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:05:37.742371 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:37.742308    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:05:38.068492 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:38.068382    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:05:47.742617 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:47.742520    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:05:48.068971 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:48.068868    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:05:57.741998 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:57.741884    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:05:58.068677 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:05:58.068577    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:06:07.741818 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:07.741768    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:06:08.068330 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:08.068231    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:06:17.742012 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:17.741967    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:06:18.068334 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:18.068231    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:06:27.742003 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:27.741951    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:06:28.068989 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:28.068892    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:06:37.742174 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:37.742113    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:06:38.068740 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:38.068635    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:06:47.742729 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:47.742675    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:06:48.069068 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:48.068954    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:06:57.741962 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:57.741898    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:06:58.068864 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:06:58.068759    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:07:00.708362 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:00.708329    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:07:00.728748 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:00.728723    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:07:07.742074 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:07.742016    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 20 11:07:08.069000 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:08.068894    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 20 11:07:17.758344 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:17.758310    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:07:17.769976 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:17.769948    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:07:18.078363 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:18.078281    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:07:18.086367 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:18.086332    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:07:26.553278 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:26.553194    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"]
Jul 20 11:07:26.555417 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:26.555390    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"]
Jul 20 11:07:41.577611 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:41.577535    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main" containerID="cri-o://a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78" gracePeriod=45
Jul 20 11:07:41.584414 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:41.584372    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main" containerID="cri-o://acf8c15c095f580ede1eb3045e0b83e1be4a168d3e7359a81ebbd9cf05e5259b" gracePeriod=45
Jul 20 11:07:42.397015 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.396960    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="llm-d-routing-sidecar" containerID="cri-o://72fb2d7a1641c8e5450c259b9c920026eb9b9e114190139058f8df0abd58b2b6" gracePeriod=45
Jul 20 11:07:42.558297 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.558270    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:07:42.618141 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.618101    2572 generic.go:356] "Generic (PLEG): container finished" podID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerID="acf8c15c095f580ede1eb3045e0b83e1be4a168d3e7359a81ebbd9cf05e5259b" exitCode=0
Jul 20 11:07:42.618141 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.618126    2572 generic.go:356] "Generic (PLEG): container finished" podID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerID="72fb2d7a1641c8e5450c259b9c920026eb9b9e114190139058f8df0abd58b2b6" exitCode=0
Jul 20 11:07:42.618628 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.618167    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" event={"ID":"bf97e47c-4717-4ed3-b2de-16b83f087651","Type":"ContainerDied","Data":"acf8c15c095f580ede1eb3045e0b83e1be4a168d3e7359a81ebbd9cf05e5259b"}
Jul 20 11:07:42.618628 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.618258    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" event={"ID":"bf97e47c-4717-4ed3-b2de-16b83f087651","Type":"ContainerDied","Data":"72fb2d7a1641c8e5450c259b9c920026eb9b9e114190139058f8df0abd58b2b6"}
Jul 20 11:07:42.619775 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.619749    2572 generic.go:356] "Generic (PLEG): container finished" podID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerID="a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78" exitCode=0
Jul 20 11:07:42.619874 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.619794    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" event={"ID":"a43a41cb-5732-4fb9-aba6-9f8c9267f74f","Type":"ContainerDied","Data":"a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78"}
Jul 20 11:07:42.619874 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.619799    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"
Jul 20 11:07:42.619874 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.619835    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x" event={"ID":"a43a41cb-5732-4fb9-aba6-9f8c9267f74f","Type":"ContainerDied","Data":"67e527919177ad3ccaa55f29f7070ba41038c824b7addce377064fadfe096d89"}
Jul 20 11:07:42.619874 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.619859    2572 scope.go:117] "RemoveContainer" containerID="a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78"
Jul 20 11:07:42.629601 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.629577    2572 scope.go:117] "RemoveContainer" containerID="bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0"
Jul 20 11:07:42.638294 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.638272    2572 scope.go:117] "RemoveContainer" containerID="a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78"
Jul 20 11:07:42.638660 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:07:42.638633    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78\": container with ID starting with a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78 not found: ID does not exist" containerID="a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78"
Jul 20 11:07:42.638747 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.638675    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78"} err="failed to get container status \"a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78\": rpc error: code = NotFound desc = could not find container \"a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78\": container with ID starting with a376531d214457326a261edbdf691c20333169912ec01c9cd5feba507901db78 not found: ID does not exist"
Jul 20 11:07:42.638747 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.638701    2572 scope.go:117] "RemoveContainer" containerID="bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0"
Jul 20 11:07:42.638997 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:07:42.638974    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0\": container with ID starting with bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0 not found: ID does not exist" containerID="bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0"
Jul 20 11:07:42.639049 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.639005    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0"} err="failed to get container status \"bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0\": rpc error: code = NotFound desc = could not find container \"bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0\": container with ID starting with bc32da0240b4e76a792ec2392ad4917fc389f591cb89d05dc4132d004af303b0 not found: ID does not exist"
Jul 20 11:07:42.647854 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.647824    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:07:42.697466 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.697418    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-dshm\") pod \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") "
Jul 20 11:07:42.697675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.697495    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-home\") pod \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") "
Jul 20 11:07:42.697675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.697530    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-model-cache\") pod \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") "
Jul 20 11:07:42.697675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.697577    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tls-certs\") pod \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") "
Jul 20 11:07:42.697675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.697609    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kserve-provision-location\") pod \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") "
Jul 20 11:07:42.697675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.697656    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chfpp\" (UniqueName: \"kubernetes.io/projected/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kube-api-access-chfpp\") pod \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") "
Jul 20 11:07:42.697971 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.697709    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tmp-dir\") pod \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\" (UID: \"a43a41cb-5732-4fb9-aba6-9f8c9267f74f\") "
Jul 20 11:07:42.697971 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.697812    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a43a41cb-5732-4fb9-aba6-9f8c9267f74f" (UID: "a43a41cb-5732-4fb9-aba6-9f8c9267f74f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.698069 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.698012    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.698260 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.698233    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-home" (OuterVolumeSpecName: "home") pod "a43a41cb-5732-4fb9-aba6-9f8c9267f74f" (UID: "a43a41cb-5732-4fb9-aba6-9f8c9267f74f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.700282 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.700255    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a43a41cb-5732-4fb9-aba6-9f8c9267f74f" (UID: "a43a41cb-5732-4fb9-aba6-9f8c9267f74f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:07:42.700396 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.700215    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kube-api-access-chfpp" (OuterVolumeSpecName: "kube-api-access-chfpp") pod "a43a41cb-5732-4fb9-aba6-9f8c9267f74f" (UID: "a43a41cb-5732-4fb9-aba6-9f8c9267f74f"). InnerVolumeSpecName "kube-api-access-chfpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:07:42.700396 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.700300    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-dshm" (OuterVolumeSpecName: "dshm") pod "a43a41cb-5732-4fb9-aba6-9f8c9267f74f" (UID: "a43a41cb-5732-4fb9-aba6-9f8c9267f74f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.712904 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.712863    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a43a41cb-5732-4fb9-aba6-9f8c9267f74f" (UID: "a43a41cb-5732-4fb9-aba6-9f8c9267f74f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.760827 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.760773    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a43a41cb-5732-4fb9-aba6-9f8c9267f74f" (UID: "a43a41cb-5732-4fb9-aba6-9f8c9267f74f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.798953 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.798907    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf97e47c-4717-4ed3-b2de-16b83f087651-tls-certs\") pod \"bf97e47c-4717-4ed3-b2de-16b83f087651\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") "
Jul 20 11:07:42.798953 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.798960    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-kserve-provision-location\") pod \"bf97e47c-4717-4ed3-b2de-16b83f087651\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") "
Jul 20 11:07:42.799202 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.798985    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj2rf\" (UniqueName: \"kubernetes.io/projected/bf97e47c-4717-4ed3-b2de-16b83f087651-kube-api-access-qj2rf\") pod \"bf97e47c-4717-4ed3-b2de-16b83f087651\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") "
Jul 20 11:07:42.799202 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799011    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-home\") pod \"bf97e47c-4717-4ed3-b2de-16b83f087651\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") "
Jul 20 11:07:42.799202 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799039    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-tmp-dir\") pod \"bf97e47c-4717-4ed3-b2de-16b83f087651\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") "
Jul 20 11:07:42.799202 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799066    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-model-cache\") pod \"bf97e47c-4717-4ed3-b2de-16b83f087651\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") "
Jul 20 11:07:42.799202 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799111    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-dshm\") pod \"bf97e47c-4717-4ed3-b2de-16b83f087651\" (UID: \"bf97e47c-4717-4ed3-b2de-16b83f087651\") "
Jul 20 11:07:42.799475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799340    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.799475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799372    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.799475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799386    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.799475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799401    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.799475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799415    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chfpp\" (UniqueName: \"kubernetes.io/projected/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-kube-api-access-chfpp\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.799475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799431    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43a41cb-5732-4fb9-aba6-9f8c9267f74f-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.799475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799403    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf97e47c-4717-4ed3-b2de-16b83f087651" (UID: "bf97e47c-4717-4ed3-b2de-16b83f087651"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.799851 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.799825    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-home" (OuterVolumeSpecName: "home") pod "bf97e47c-4717-4ed3-b2de-16b83f087651" (UID: "bf97e47c-4717-4ed3-b2de-16b83f087651"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.801391 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.801358    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf97e47c-4717-4ed3-b2de-16b83f087651-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf97e47c-4717-4ed3-b2de-16b83f087651" (UID: "bf97e47c-4717-4ed3-b2de-16b83f087651"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:07:42.801503 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.801386    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf97e47c-4717-4ed3-b2de-16b83f087651-kube-api-access-qj2rf" (OuterVolumeSpecName: "kube-api-access-qj2rf") pod "bf97e47c-4717-4ed3-b2de-16b83f087651" (UID: "bf97e47c-4717-4ed3-b2de-16b83f087651"). InnerVolumeSpecName "kube-api-access-qj2rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:07:42.801666 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.801634    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-dshm" (OuterVolumeSpecName: "dshm") pod "bf97e47c-4717-4ed3-b2de-16b83f087651" (UID: "bf97e47c-4717-4ed3-b2de-16b83f087651"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.812047 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.812006    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf97e47c-4717-4ed3-b2de-16b83f087651" (UID: "bf97e47c-4717-4ed3-b2de-16b83f087651"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.860451 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.860342    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf97e47c-4717-4ed3-b2de-16b83f087651" (UID: "bf97e47c-4717-4ed3-b2de-16b83f087651"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:42.900060 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.900030    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.900060 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.900060    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.900210 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.900073    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.900210 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.900080    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.900210 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.900088    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf97e47c-4717-4ed3-b2de-16b83f087651-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.900210 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.900096    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf97e47c-4717-4ed3-b2de-16b83f087651-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.900210 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.900105    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj2rf\" (UniqueName: \"kubernetes.io/projected/bf97e47c-4717-4ed3-b2de-16b83f087651-kube-api-access-qj2rf\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:07:42.943406 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.943371    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"]
Jul 20 11:07:42.947373 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:42.947342    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-587nk8x"]
Jul 20 11:07:43.625546 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:43.625443    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"
Jul 20 11:07:43.625546 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:43.625448    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh" event={"ID":"bf97e47c-4717-4ed3-b2de-16b83f087651","Type":"ContainerDied","Data":"9cc33d74e59784057a8b114d0cd0a653a0a9a9492e1d8f9b0a9ca67bf08d0f5a"}
Jul 20 11:07:43.626081 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:43.625601    2572 scope.go:117] "RemoveContainer" containerID="acf8c15c095f580ede1eb3045e0b83e1be4a168d3e7359a81ebbd9cf05e5259b"
Jul 20 11:07:43.633934 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:43.633918    2572 scope.go:117] "RemoveContainer" containerID="3458300fe85a09b1c3ff781db656665195635cc4a56c70ee38a011cbd1347d00"
Jul 20 11:07:43.641871 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:43.641854    2572 scope.go:117] "RemoveContainer" containerID="72fb2d7a1641c8e5450c259b9c920026eb9b9e114190139058f8df0abd58b2b6"
Jul 20 11:07:43.649765 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:43.649742    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"]
Jul 20 11:07:43.654368 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:43.654345    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5cbcfc79kxcjh"]
Jul 20 11:07:44.519918 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:44.519887    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" path="/var/lib/kubelet/pods/a43a41cb-5732-4fb9-aba6-9f8c9267f74f/volumes"
Jul 20 11:07:44.520305 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:44.520292    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" path="/var/lib/kubelet/pods/bf97e47c-4717-4ed3-b2de-16b83f087651/volumes"
Jul 20 11:07:51.225294 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225257    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"]
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225539    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225549    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225584    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="storage-initializer"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225590    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="storage-initializer"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225601    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="llm-d-routing-sidecar"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225606    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="llm-d-routing-sidecar"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225616    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225621    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225631    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="storage-initializer"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225638    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="storage-initializer"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225683    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43a41cb-5732-4fb9-aba6-9f8c9267f74f" containerName="main"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225692    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="main"
Jul 20 11:07:51.225745 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.225698    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf97e47c-4717-4ed3-b2de-16b83f087651" containerName="llm-d-routing-sidecar"
Jul 20 11:07:51.228527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.228510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.232087 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.232061    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Jul 20 11:07:51.232302 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.232080    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-ntncw\""
Jul 20 11:07:51.232466 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.232153    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Jul 20 11:07:51.232597 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.232161    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Jul 20 11:07:51.232597 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.232283    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-h5mmw\""
Jul 20 11:07:51.233128 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.233106    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 20 11:07:51.244704 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.244671    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"]
Jul 20 11:07:51.256533 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.256506    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"]
Jul 20 11:07:51.257685 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.257656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-model-cache\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.257813 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.257699    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-home\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.257813 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.257744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-dshm\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.257813 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.257782    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.257813 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.257807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9fcab0-840a-4a04-b94b-055d20c0c828-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.258067 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.257847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.258067 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.257881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pkb\" (UniqueName: \"kubernetes.io/projected/ae9fcab0-840a-4a04-b94b-055d20c0c828-kube-api-access-94pkb\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.259779 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.259765    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.271972 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.271940    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"]
Jul 20 11:07:51.358232 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358232 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358258    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9fcab0-840a-4a04-b94b-055d20c0c828-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358285    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358313    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkd9h\" (UniqueName: \"kubernetes.io/projected/955457d9-d822-4a32-a59b-0588452234df-kube-api-access-dkd9h\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358338    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pkb\" (UniqueName: \"kubernetes.io/projected/ae9fcab0-840a-4a04-b94b-055d20c0c828-kube-api-access-94pkb\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358354    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358377    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358412    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.358461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358462    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-model-cache\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358902 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358486    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-home\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358902 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358617    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955457d9-d822-4a32-a59b-0588452234df-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.358902 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358665    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-dshm\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358902 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358670    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358902 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358738    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358902 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358768    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-home\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.358902 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.358829    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-model-cache\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.360641 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.360621    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-dshm\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.360833 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.360815    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9fcab0-840a-4a04-b94b-055d20c0c828-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.370899 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.370869    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pkb\" (UniqueName: \"kubernetes.io/projected/ae9fcab0-840a-4a04-b94b-055d20c0c828-kube-api-access-94pkb\") pod \"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.460094 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460049    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkd9h\" (UniqueName: \"kubernetes.io/projected/955457d9-d822-4a32-a59b-0588452234df-kube-api-access-dkd9h\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460299 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460299 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460146    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460299 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460167    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460299 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460229    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460299 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460294    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955457d9-d822-4a32-a59b-0588452234df-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460339    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460679 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460628    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460679 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460663    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460757 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460730    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.460833 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.460813    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.462703 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.462675    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.462942 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.462925    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955457d9-d822-4a32-a59b-0588452234df-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.467982 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.467954    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkd9h\" (UniqueName: \"kubernetes.io/projected/955457d9-d822-4a32-a59b-0588452234df-kube-api-access-dkd9h\") pod \"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.539340 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.539240    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:51.570994 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.570967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:07:51.705754 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.705721    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"]
Jul 20 11:07:51.708978 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:07:51.708953    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9fcab0_840a_4a04_b94b_055d20c0c828.slice/crio-4c65b4cbe0f7f3bc978fdc73a353ff9a63fd8dbcbe03ccd4295d8fa09e17e932 WatchSource:0}: Error finding container 4c65b4cbe0f7f3bc978fdc73a353ff9a63fd8dbcbe03ccd4295d8fa09e17e932: Status 404 returned error can't find the container with id 4c65b4cbe0f7f3bc978fdc73a353ff9a63fd8dbcbe03ccd4295d8fa09e17e932
Jul 20 11:07:51.724392 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:51.724364    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"]
Jul 20 11:07:51.726434 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:07:51.726408    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955457d9_d822_4a32_a59b_0588452234df.slice/crio-8f332c79544c4cd221ac77db2bf96f0fef4116b634360b0ac4b65c3aebe411aa WatchSource:0}: Error finding container 8f332c79544c4cd221ac77db2bf96f0fef4116b634360b0ac4b65c3aebe411aa: Status 404 returned error can't find the container with id 8f332c79544c4cd221ac77db2bf96f0fef4116b634360b0ac4b65c3aebe411aa
Jul 20 11:07:52.664458 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:52.664417    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" event={"ID":"955457d9-d822-4a32-a59b-0588452234df","Type":"ContainerStarted","Data":"6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702"}
Jul 20 11:07:52.665089 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:52.664483    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" event={"ID":"955457d9-d822-4a32-a59b-0588452234df","Type":"ContainerStarted","Data":"8f332c79544c4cd221ac77db2bf96f0fef4116b634360b0ac4b65c3aebe411aa"}
Jul 20 11:07:52.666289 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:52.666250    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" event={"ID":"ae9fcab0-840a-4a04-b94b-055d20c0c828","Type":"ContainerStarted","Data":"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144"}
Jul 20 11:07:52.666412 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:52.666300    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" event={"ID":"ae9fcab0-840a-4a04-b94b-055d20c0c828","Type":"ContainerStarted","Data":"4c65b4cbe0f7f3bc978fdc73a353ff9a63fd8dbcbe03ccd4295d8fa09e17e932"}
Jul 20 11:07:52.666412 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:52.666334    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:07:53.675600 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:53.675504    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" event={"ID":"ae9fcab0-840a-4a04-b94b-055d20c0c828","Type":"ContainerStarted","Data":"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce"}
Jul 20 11:07:55.688180 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:55.688143    2572 generic.go:356] "Generic (PLEG): container finished" podID="955457d9-d822-4a32-a59b-0588452234df" containerID="6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702" exitCode=0
Jul 20 11:07:55.688670 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:55.688204    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" event={"ID":"955457d9-d822-4a32-a59b-0588452234df","Type":"ContainerDied","Data":"6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702"}
Jul 20 11:07:56.695987 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:56.695945    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" event={"ID":"955457d9-d822-4a32-a59b-0588452234df","Type":"ContainerStarted","Data":"257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5"}
Jul 20 11:07:56.722673 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:56.722599    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podStartSLOduration=2.362515888 podStartE2EDuration="5.722578495s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.360062607s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:07:56.71910277 +0000 UTC m=+4256.732204661" watchObservedRunningTime="2026-07-20 11:07:56.722578495 +0000 UTC m=+4256.735680393"
Jul 20 11:07:57.701871 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:57.701834    2572 generic.go:356] "Generic (PLEG): container finished" podID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerID="361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce" exitCode=0
Jul 20 11:07:57.702343 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:57.701899    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" event={"ID":"ae9fcab0-840a-4a04-b94b-055d20c0c828","Type":"ContainerDied","Data":"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce"}
Jul 20 11:07:58.711618 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:58.711582    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" event={"ID":"ae9fcab0-840a-4a04-b94b-055d20c0c828","Type":"ContainerStarted","Data":"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26"}
Jul 20 11:07:58.739568 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:07:58.739483    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podStartSLOduration=3.53910117 podStartE2EDuration="7.739461683s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.200360513s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:07:58.73715188 +0000 UTC m=+4258.750253783" watchObservedRunningTime="2026-07-20 11:07:58.739461683 +0000 UTC m=+4258.752563580"
Jul 20 11:08:01.540280 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:01.540238    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:08:01.540805 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:01.540308    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:08:01.541965 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:01.541926    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:08:01.558631 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:01.558604    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:08:01.571880 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:01.571839    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:08:01.572074 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:01.571914    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:08:01.573446 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:01.573412    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:08:11.540045 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:11.539987    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:08:11.572001 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:11.571953    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:08:21.539891 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:21.539838    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:08:21.571975 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:21.571933    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:08:31.540337 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:31.540283    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:08:31.571837 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:31.571795    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:08:41.540095 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:41.540044    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:08:41.571798 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:41.571752    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:08:51.540239 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:51.540179    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:08:51.571572 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:08:51.571517    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:09:01.539983 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:01.539929    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:09:01.572060 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:01.572018    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:09:11.540665 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:11.540598    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:09:11.571688 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:11.571637    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:09:21.540281 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:21.540214    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:09:21.571643 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:21.571592    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:09:31.540103 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:31.540060    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:09:31.571615 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:31.571575    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:09:41.539948 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:41.539901    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:09:41.571512 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:41.571458    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:09:51.540075 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:51.540032    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:09:51.571828 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:09:51.571786    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:10:01.540437 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:01.540391    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:10:01.572331 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:01.572282    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:10:08.837287 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:08.837241    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:05Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:05Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:05Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:05Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:08.880986 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:08.880933    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:08.913240 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:08.913187    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:08.950263 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:08.950208    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:08.972416 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:08.972369    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:08Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:08.972416 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:08.972407    2572 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 20 11:10:11.540422 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:11.540361    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:10:11.572250 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:11.572201    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:10:19.154152 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:19.154092    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:19.175502 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:19.175450    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:19.194296 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:19.194250    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:19.213363 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:19.213314    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:19.237528 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:19.237480    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:19Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:19.237528 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:19.237521    2572 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 20 11:10:21.543926 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:21.543871    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:10:21.571863 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:21.571816    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:10:31.540339 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:31.540290    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Jul 20 11:10:31.571878 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:31.571839    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 20 11:10:39.845228 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:39.845180    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:39.864624 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:39.864580    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:39.880617 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:39.880576    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:39.896937 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:39.896903    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:39.913052 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:39.913017    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:39Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-138-196.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:39.913052 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:39.913050    2572 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 20 11:10:41.558775 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:41.558746    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:10:41.581470 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:41.581441    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:10:41.589109 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:41.589081    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:10:41.629138 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:41.629075    2572 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ae9fcab0-840a-4a04-b94b-055d20c0c828\\\"},\\\"status\\\":{\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"},\\\"containerID\\\":\\\"cri-o://22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26\\\",\\\"image\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0\\\",\\\"imageID\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"2\\\",\\\"memory\\\":\\\"7Gi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-20T11:07:57Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1001130000],\\\"uid\\\":1001130000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/mnt/models\\\",\\\"name\\\":\\\"kserve-provision-location\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-94pkb\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/pod?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:10:41.649738 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:41.649687    2572 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ae9fcab0-840a-4a04-b94b-055d20c0c828\\\"},\\\"status\\\":{\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"},\\\"containerID\\\":\\\"cri-o://22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26\\\",\\\"image\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0\\\",\\\"imageID\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"2\\\",\\\"memory\\\":\\\"7Gi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-20T11:07:57Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1001130000],\\\"uid\\\":1001130000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/mnt/models\\\",\\\"name\\\":\\\"kserve-provision-location\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-94pkb\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/pod?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:10:41.665368 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:41.665316    2572 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"955457d9-d822-4a32-a59b-0588452234df\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-20T11:10:41Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-20T11:10:41Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"},\\\"containerID\\\":\\\"cri-o://257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5\\\",\\\"image\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0\\\",\\\"imageID\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"2\\\",\\\"memory\\\":\\\"7Gi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-20T11:07:55Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1001130000],\\\"uid\\\":1001130000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/mnt/models\\\",\\\"name\\\":\\\"kserve-provision-location\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-dkd9h\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/pod?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:10:42.467779 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:42.467749    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:10:42.487488 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:42.487433    2572 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ae9fcab0-840a-4a04-b94b-055d20c0c828\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"},\\\"containerID\\\":\\\"cri-o://22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26\\\",\\\"image\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0\\\",\\\"imageID\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"2\\\",\\\"memory\\\":\\\"7Gi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-20T11:07:57Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1001130000],\\\"uid\\\":1001130000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/mnt/models\\\",\\\"name\\\":\\\"kserve-provision-location\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-94pkb\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/pod?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:10:42.505350 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:10:42.505302    2572 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"955457d9-d822-4a32-a59b-0588452234df\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-20T11:10:41Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-20T11:10:41Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"},\\\"containerID\\\":\\\"cri-o://257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5\\\",\\\"image\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0\\\",\\\"imageID\\\":\\\"public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"2\\\",\\\"memory\\\":\\\"7Gi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"2Gi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-20T11:07:55Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1001130000],\\\"uid\\\":1001130000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/mnt/models\\\",\\\"name\\\":\\\"kserve-provision-location\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-dkd9h\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/pod?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:10:59.466461 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:59.466426    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"]
Jul 20 11:10:59.468004 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:10:59.467977    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"]
Jul 20 11:11:11.828427 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:11.828387    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-q2c7x"]
Jul 20 11:11:11.832892 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:11.832873    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:11.836071 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:11.836047    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\""
Jul 20 11:11:11.907644 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:11.907603    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-ready\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:11.907644 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:11.907652    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twdjs\" (UniqueName: \"kubernetes.io/projected/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-kube-api-access-twdjs\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:11.907895 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:11.907679    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:11.907895 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:11.907770    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.008677 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.008640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twdjs\" (UniqueName: \"kubernetes.io/projected/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-kube-api-access-twdjs\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.008876 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.008694    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.008876 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.008744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.008876 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.008828    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.009047 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.008871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-ready\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.009145 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.009128    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-ready\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.009294 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.009276    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.017077 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.017049    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twdjs\" (UniqueName: \"kubernetes.io/projected/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-kube-api-access-twdjs\") pod \"cni-sysctl-allowlist-ds-q2c7x\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.142329 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.142225    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.154096 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:11:12.154066    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5058bd63_cc3b_45ea_bf1d_30b5b29b0d1a.slice/crio-294ed6f126cb37f93bb1d446cf233c8b0ff8825a4e3edc95a8834a8326ade9a8 WatchSource:0}: Error finding container 294ed6f126cb37f93bb1d446cf233c8b0ff8825a4e3edc95a8834a8326ade9a8: Status 404 returned error can't find the container with id 294ed6f126cb37f93bb1d446cf233c8b0ff8825a4e3edc95a8834a8326ade9a8
Jul 20 11:11:12.579566 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.579519    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" event={"ID":"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a","Type":"ContainerStarted","Data":"65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343"}
Jul 20 11:11:12.579752 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.579586    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" event={"ID":"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a","Type":"ContainerStarted","Data":"294ed6f126cb37f93bb1d446cf233c8b0ff8825a4e3edc95a8834a8326ade9a8"}
Jul 20 11:11:12.579752 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.579740    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:12.598465 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:12.598407    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" podStartSLOduration=1.598390443 podStartE2EDuration="1.598390443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 11:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:11:12.596915739 +0000 UTC m=+4452.610017635" watchObservedRunningTime="2026-07-20 11:11:12.598390443 +0000 UTC m=+4452.611492339"
Jul 20 11:11:13.594463 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:13.594432    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:14.491216 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:14.491144    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main" containerID="cri-o://22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26" gracePeriod=45
Jul 20 11:11:14.491936 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:14.491743    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main" containerID="cri-o://257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5" gracePeriod=45
Jul 20 11:11:14.849457 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:14.849361    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-q2c7x"]
Jul 20 11:11:15.308925 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.308890    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="llm-d-routing-sidecar" containerID="cri-o://3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144" gracePeriod=45
Jul 20 11:11:15.446582 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.446541    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:11:15.533039 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533006    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955457d9-d822-4a32-a59b-0588452234df-tls-certs\") pod \"955457d9-d822-4a32-a59b-0588452234df\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") "
Jul 20 11:11:15.533245 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533051    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-kserve-provision-location\") pod \"955457d9-d822-4a32-a59b-0588452234df\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") "
Jul 20 11:11:15.533245 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533073    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-home\") pod \"955457d9-d822-4a32-a59b-0588452234df\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") "
Jul 20 11:11:15.533245 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533094    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-tmp-dir\") pod \"955457d9-d822-4a32-a59b-0588452234df\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") "
Jul 20 11:11:15.533245 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533124    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-dshm\") pod \"955457d9-d822-4a32-a59b-0588452234df\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") "
Jul 20 11:11:15.533245 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533153    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkd9h\" (UniqueName: \"kubernetes.io/projected/955457d9-d822-4a32-a59b-0588452234df-kube-api-access-dkd9h\") pod \"955457d9-d822-4a32-a59b-0588452234df\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") "
Jul 20 11:11:15.533245 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533209    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-model-cache\") pod \"955457d9-d822-4a32-a59b-0588452234df\" (UID: \"955457d9-d822-4a32-a59b-0588452234df\") "
Jul 20 11:11:15.533977 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533716    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-model-cache" (OuterVolumeSpecName: "model-cache") pod "955457d9-d822-4a32-a59b-0588452234df" (UID: "955457d9-d822-4a32-a59b-0588452234df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.533977 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.533832    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-home" (OuterVolumeSpecName: "home") pod "955457d9-d822-4a32-a59b-0588452234df" (UID: "955457d9-d822-4a32-a59b-0588452234df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.538612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.536723    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-dshm" (OuterVolumeSpecName: "dshm") pod "955457d9-d822-4a32-a59b-0588452234df" (UID: "955457d9-d822-4a32-a59b-0588452234df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.538612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.536889    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955457d9-d822-4a32-a59b-0588452234df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "955457d9-d822-4a32-a59b-0588452234df" (UID: "955457d9-d822-4a32-a59b-0588452234df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:11:15.538612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.536914    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955457d9-d822-4a32-a59b-0588452234df-kube-api-access-dkd9h" (OuterVolumeSpecName: "kube-api-access-dkd9h") pod "955457d9-d822-4a32-a59b-0588452234df" (UID: "955457d9-d822-4a32-a59b-0588452234df"). InnerVolumeSpecName "kube-api-access-dkd9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:11:15.548026 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.547987    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "955457d9-d822-4a32-a59b-0588452234df" (UID: "955457d9-d822-4a32-a59b-0588452234df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.569845 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.569820    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:11:15.595599 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.595540    2572 generic.go:356] "Generic (PLEG): container finished" podID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerID="22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26" exitCode=0
Jul 20 11:11:15.595599 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.595593    2572 generic.go:356] "Generic (PLEG): container finished" podID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerID="3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144" exitCode=0
Jul 20 11:11:15.595822 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.595632    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"
Jul 20 11:11:15.595822 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.595635    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" event={"ID":"ae9fcab0-840a-4a04-b94b-055d20c0c828","Type":"ContainerDied","Data":"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26"}
Jul 20 11:11:15.595822 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.595735    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" event={"ID":"ae9fcab0-840a-4a04-b94b-055d20c0c828","Type":"ContainerDied","Data":"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144"}
Jul 20 11:11:15.595822 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.595761    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx" event={"ID":"ae9fcab0-840a-4a04-b94b-055d20c0c828","Type":"ContainerDied","Data":"4c65b4cbe0f7f3bc978fdc73a353ff9a63fd8dbcbe03ccd4295d8fa09e17e932"}
Jul 20 11:11:15.595822 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.595793    2572 scope.go:117] "RemoveContainer" containerID="22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26"
Jul 20 11:11:15.597289 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.597265    2572 generic.go:356] "Generic (PLEG): container finished" podID="955457d9-d822-4a32-a59b-0588452234df" containerID="257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5" exitCode=0
Jul 20 11:11:15.597396 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.597328    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"
Jul 20 11:11:15.597472 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.597404    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" event={"ID":"955457d9-d822-4a32-a59b-0588452234df","Type":"ContainerDied","Data":"257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5"}
Jul 20 11:11:15.597472 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.597460    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6" event={"ID":"955457d9-d822-4a32-a59b-0588452234df","Type":"ContainerDied","Data":"8f332c79544c4cd221ac77db2bf96f0fef4116b634360b0ac4b65c3aebe411aa"}
Jul 20 11:11:15.597662 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.597627    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" podUID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" gracePeriod=30
Jul 20 11:11:15.609376 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.609348    2572 scope.go:117] "RemoveContainer" containerID="361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce"
Jul 20 11:11:15.611831 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.611804    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "955457d9-d822-4a32-a59b-0588452234df" (UID: "955457d9-d822-4a32-a59b-0588452234df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.619385 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.619369    2572 scope.go:117] "RemoveContainer" containerID="3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144"
Jul 20 11:11:15.627277 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.627254    2572 scope.go:117] "RemoveContainer" containerID="22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26"
Jul 20 11:11:15.627543 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:15.627521    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26\": container with ID starting with 22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26 not found: ID does not exist" containerID="22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26"
Jul 20 11:11:15.627631 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.627575    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26"} err="failed to get container status \"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26\": rpc error: code = NotFound desc = could not find container \"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26\": container with ID starting with 22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26 not found: ID does not exist"
Jul 20 11:11:15.627631 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.627595    2572 scope.go:117] "RemoveContainer" containerID="361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce"
Jul 20 11:11:15.627837 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:15.627816    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce\": container with ID starting with 361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce not found: ID does not exist" containerID="361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce"
Jul 20 11:11:15.627912 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.627849    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce"} err="failed to get container status \"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce\": rpc error: code = NotFound desc = could not find container \"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce\": container with ID starting with 361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce not found: ID does not exist"
Jul 20 11:11:15.627912 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.627872    2572 scope.go:117] "RemoveContainer" containerID="3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144"
Jul 20 11:11:15.628178 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:15.628152    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144\": container with ID starting with 3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144 not found: ID does not exist" containerID="3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144"
Jul 20 11:11:15.628225 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.628187    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144"} err="failed to get container status \"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144\": rpc error: code = NotFound desc = could not find container \"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144\": container with ID starting with 3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144 not found: ID does not exist"
Jul 20 11:11:15.628225 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.628208    2572 scope.go:117] "RemoveContainer" containerID="22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26"
Jul 20 11:11:15.628456 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.628434    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26"} err="failed to get container status \"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26\": rpc error: code = NotFound desc = could not find container \"22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26\": container with ID starting with 22753359dac91d0c5309d3acf7f8d8714fa18ff3964cf676cf5f2530eb2e6f26 not found: ID does not exist"
Jul 20 11:11:15.628518 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.628456    2572 scope.go:117] "RemoveContainer" containerID="361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce"
Jul 20 11:11:15.628678 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.628655    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce"} err="failed to get container status \"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce\": rpc error: code = NotFound desc = could not find container \"361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce\": container with ID starting with 361447eec46718bdb831c121f1b3374c22eb239563b689fe18513ddc82c7b2ce not found: ID does not exist"
Jul 20 11:11:15.628739 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.628681    2572 scope.go:117] "RemoveContainer" containerID="3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144"
Jul 20 11:11:15.628936 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.628909    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144"} err="failed to get container status \"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144\": rpc error: code = NotFound desc = could not find container \"3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144\": container with ID starting with 3e2c0f9318a7e9ae1fd1e23ef07664b4f6ab9ca2bff145ade0cfd911b4671144 not found: ID does not exist"
Jul 20 11:11:15.628936 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.628930    2572 scope.go:117] "RemoveContainer" containerID="257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5"
Jul 20 11:11:15.634060 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634038    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-home\") pod \"ae9fcab0-840a-4a04-b94b-055d20c0c828\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") "
Jul 20 11:11:15.634150 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634080    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-model-cache\") pod \"ae9fcab0-840a-4a04-b94b-055d20c0c828\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") "
Jul 20 11:11:15.634150 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634116    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-kserve-provision-location\") pod \"ae9fcab0-840a-4a04-b94b-055d20c0c828\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") "
Jul 20 11:11:15.634150 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634139    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-tmp-dir\") pod \"ae9fcab0-840a-4a04-b94b-055d20c0c828\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") "
Jul 20 11:11:15.634307 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634164    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94pkb\" (UniqueName: \"kubernetes.io/projected/ae9fcab0-840a-4a04-b94b-055d20c0c828-kube-api-access-94pkb\") pod \"ae9fcab0-840a-4a04-b94b-055d20c0c828\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") "
Jul 20 11:11:15.634307 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634204    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9fcab0-840a-4a04-b94b-055d20c0c828-tls-certs\") pod \"ae9fcab0-840a-4a04-b94b-055d20c0c828\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") "
Jul 20 11:11:15.634307 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634237    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-dshm\") pod \"ae9fcab0-840a-4a04-b94b-055d20c0c828\" (UID: \"ae9fcab0-840a-4a04-b94b-055d20c0c828\") "
Jul 20 11:11:15.634463 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634389    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae9fcab0-840a-4a04-b94b-055d20c0c828" (UID: "ae9fcab0-840a-4a04-b94b-055d20c0c828"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.634524 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634484    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.634524 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634505    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.634650 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634543    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.634650 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634582    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.634650 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634594    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.634650 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634607    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkd9h\" (UniqueName: \"kubernetes.io/projected/955457d9-d822-4a32-a59b-0588452234df-kube-api-access-dkd9h\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.634650 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634619    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955457d9-d822-4a32-a59b-0588452234df-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.634650 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634633    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955457d9-d822-4a32-a59b-0588452234df-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.634954 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.634782    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-home" (OuterVolumeSpecName: "home") pod "ae9fcab0-840a-4a04-b94b-055d20c0c828" (UID: "ae9fcab0-840a-4a04-b94b-055d20c0c828"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.636831 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.636805    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-dshm" (OuterVolumeSpecName: "dshm") pod "ae9fcab0-840a-4a04-b94b-055d20c0c828" (UID: "ae9fcab0-840a-4a04-b94b-055d20c0c828"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.636933 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.636878    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9fcab0-840a-4a04-b94b-055d20c0c828-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae9fcab0-840a-4a04-b94b-055d20c0c828" (UID: "ae9fcab0-840a-4a04-b94b-055d20c0c828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:11:15.636933 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.636921    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9fcab0-840a-4a04-b94b-055d20c0c828-kube-api-access-94pkb" (OuterVolumeSpecName: "kube-api-access-94pkb") pod "ae9fcab0-840a-4a04-b94b-055d20c0c828" (UID: "ae9fcab0-840a-4a04-b94b-055d20c0c828"). InnerVolumeSpecName "kube-api-access-94pkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:11:15.637443 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.637429    2572 scope.go:117] "RemoveContainer" containerID="6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702"
Jul 20 11:11:15.645296 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.645275    2572 scope.go:117] "RemoveContainer" containerID="257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5"
Jul 20 11:11:15.645603 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:15.645585    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5\": container with ID starting with 257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5 not found: ID does not exist" containerID="257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5"
Jul 20 11:11:15.645655 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.645614    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5"} err="failed to get container status \"257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5\": rpc error: code = NotFound desc = could not find container \"257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5\": container with ID starting with 257609b0ee1b1bf276b35952873eef470be2b54b08a02bb9343bdc790850b6f5 not found: ID does not exist"
Jul 20 11:11:15.645655 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.645632    2572 scope.go:117] "RemoveContainer" containerID="6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702"
Jul 20 11:11:15.645905 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:15.645886    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702\": container with ID starting with 6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702 not found: ID does not exist" containerID="6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702"
Jul 20 11:11:15.645961 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.645910    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702"} err="failed to get container status \"6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702\": rpc error: code = NotFound desc = could not find container \"6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702\": container with ID starting with 6cf43a16e747efb10c16ad53cb1258e47cf649035c13b6e071c191521eef5702 not found: ID does not exist"
Jul 20 11:11:15.652775 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.652746    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae9fcab0-840a-4a04-b94b-055d20c0c828" (UID: "ae9fcab0-840a-4a04-b94b-055d20c0c828"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.701392 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.701345    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae9fcab0-840a-4a04-b94b-055d20c0c828" (UID: "ae9fcab0-840a-4a04-b94b-055d20c0c828"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:15.735016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.734932    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9fcab0-840a-4a04-b94b-055d20c0c828-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.735016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.734959    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.735016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.734968    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.735016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.734976    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.735016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.734985    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9fcab0-840a-4a04-b94b-055d20c0c828-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.735016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.734996    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pkb\" (UniqueName: \"kubernetes.io/projected/ae9fcab0-840a-4a04-b94b-055d20c0c828-kube-api-access-94pkb\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:15.923300 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.923269    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"]
Jul 20 11:11:15.926063 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.926018    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-bdfb7848d-7j7jx"]
Jul 20 11:11:15.936070 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.936042    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"]
Jul 20 11:11:15.941677 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:15.941651    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66c7447bc9-nx5j6"]
Jul 20 11:11:16.519422 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:16.519337    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955457d9-d822-4a32-a59b-0588452234df" path="/var/lib/kubelet/pods/955457d9-d822-4a32-a59b-0588452234df/volumes"
Jul 20 11:11:16.519813 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:16.519797    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" path="/var/lib/kubelet/pods/ae9fcab0-840a-4a04-b94b-055d20c0c828/volumes"
Jul 20 11:11:19.793782 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.793748    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"]
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794054    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="storage-initializer"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794068    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="storage-initializer"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794077    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="storage-initializer"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794083    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="storage-initializer"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794093    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="llm-d-routing-sidecar"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794099    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="llm-d-routing-sidecar"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794122    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794128    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794136    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794141    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794190    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="llm-d-routing-sidecar"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794202    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="955457d9-d822-4a32-a59b-0588452234df" containerName="main"
Jul 20 11:11:19.794257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.794208    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9fcab0-840a-4a04-b94b-055d20c0c828" containerName="main"
Jul 20 11:11:19.798161 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.798141    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.800804 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.800697    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-nlf4t\""
Jul 20 11:11:19.800804 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.800733    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Jul 20 11:11:19.800804 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.800768    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Jul 20 11:11:19.801085 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.800833    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Jul 20 11:11:19.817500 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.817474    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"]
Jul 20 11:11:19.866427 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866390    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-envoy\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.866427 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-certs\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.866759 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866454    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-data\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.866759 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866476    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-podinfo\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.866759 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-credential-socket\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.866759 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866638    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-socket\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.866759 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866717    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96hmp\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-kube-api-access-96hmp\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.866759 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2c5b1-d360-4c03-93e5-654c21ff2082-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.866968 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.866777    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-token\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.967864 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.967827    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-certs\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.967864 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.967865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-data\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968110 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.967890    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-podinfo\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968110 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.967913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-credential-socket\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968110 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968035    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-socket\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968271 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968118    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96hmp\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-kube-api-access-96hmp\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968271 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968150    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2c5b1-d360-4c03-93e5-654c21ff2082-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968271 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-token\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968271 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968205    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-envoy\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968460 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968272    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-data\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968460 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968347    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-credential-socket\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968460 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968370    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-certs\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968605 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968591    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-socket\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.968867 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.968844    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2c5b1-d360-4c03-93e5-654c21ff2082-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.970491 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.970469    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-envoy\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.970612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.970541    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-podinfo\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.978951 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.978930    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96hmp\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-kube-api-access-96hmp\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:19.979294 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:19.979274    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-token\") pod \"router-gateway-2-openshift-default-66786c46c-tpkdv\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:20.108798 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:20.108700    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:20.252472 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:20.252446    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"]
Jul 20 11:11:20.254913 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:11:20.254880    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f2c5b1_d360_4c03_93e5_654c21ff2082.slice/crio-5f92e09574120fbde0d06d8f3fd81d436c8eb936fa515f08b86c9fd339c744c6 WatchSource:0}: Error finding container 5f92e09574120fbde0d06d8f3fd81d436c8eb936fa515f08b86c9fd339c744c6: Status 404 returned error can't find the container with id 5f92e09574120fbde0d06d8f3fd81d436c8eb936fa515f08b86c9fd339c744c6
Jul 20 11:11:20.257001 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:20.256967    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 11:11:20.257105 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:20.257050    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 11:11:20.257171 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:20.257101    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 11:11:20.622580 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:20.622514    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" event={"ID":"41f2c5b1-d360-4c03-93e5-654c21ff2082","Type":"ContainerStarted","Data":"cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7"}
Jul 20 11:11:20.622749 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:20.622606    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" event={"ID":"41f2c5b1-d360-4c03-93e5-654c21ff2082","Type":"ContainerStarted","Data":"5f92e09574120fbde0d06d8f3fd81d436c8eb936fa515f08b86c9fd339c744c6"}
Jul 20 11:11:20.655396 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:20.655311    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" podStartSLOduration=1.655294231 podStartE2EDuration="1.655294231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:11:20.654596211 +0000 UTC m=+4460.667698108" watchObservedRunningTime="2026-07-20 11:11:20.655294231 +0000 UTC m=+4460.668396128"
Jul 20 11:11:21.109582 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:21.109524    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:21.114600 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:21.114568    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:21.626490 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:21.626459    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:21.627602 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:21.627579    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:23.585940 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:23.585894    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.586873 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:23.586844    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.587734 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:23.587707    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.587790 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:23.587747    2572 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" podUID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:32.686163 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:32.686125    2572 generic.go:356] "Generic (PLEG): container finished" podID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerID="cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7" exitCode=0
Jul 20 11:11:32.686633 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:32.686200    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" event={"ID":"41f2c5b1-d360-4c03-93e5-654c21ff2082","Type":"ContainerDied","Data":"cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7"}
Jul 20 11:11:32.686699 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:32.686680    2572 scope.go:117] "RemoveContainer" containerID="cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7"
Jul 20 11:11:32.687632 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:32.687602    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 11:11:32.687726 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:32.687669    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 11:11:32.687726 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:32.687701    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 11:11:33.585951 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:33.585904    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.586904 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:33.586859    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.587803 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:33.587774    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.587895 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:33.587812    2572 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" podUID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:33.691723 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:33.691687    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" event={"ID":"41f2c5b1-d360-4c03-93e5-654c21ff2082","Type":"ContainerStarted","Data":"cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10"}
Jul 20 11:11:34.109314 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:34.109276    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:34.110455 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:34.110420    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 20 11:11:35.110022 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:35.109975    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 20 11:11:36.109479 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:36.109434    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 20 11:11:36.626811 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:36.626776    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:37.109474 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:37.109426    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 20 11:11:38.109718 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:38.109671    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 20 11:11:39.113082 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:39.113049    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:39.114067 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:39.114046    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:11:43.585747 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:43.585701    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.586692 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:43.586660    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.587600 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:43.587570    2572 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.587657 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:43.587615    2572 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" podUID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:45.332662 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.332616    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"]
Jul 20 11:11:45.337294 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.337260    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.340691 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.340627    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Jul 20 11:11:45.341207 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.341046    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-s7dcb\""
Jul 20 11:11:45.341207 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.341108    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-tqhrb\""
Jul 20 11:11:45.341456 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.341431    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 20 11:11:45.350319 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.350285    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"]
Jul 20 11:11:45.362855 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.362829    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"]
Jul 20 11:11:45.365832 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.365814    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.379646 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.379621    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"]
Jul 20 11:11:45.509083 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509042    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.509083 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509081    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.509313 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509105    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7d326-5ac1-45b3-a6b1-ba350765852c-tls-certs\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.509313 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.509313 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509230    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwpc\" (UniqueName: \"kubernetes.io/projected/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kube-api-access-5hwpc\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.509313 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509253    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-dshm\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.509313 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509277    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.509313 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509293    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-model-cache\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.509510 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.509510 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509405    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-home\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.509510 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509432    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ls8\" (UniqueName: \"kubernetes.io/projected/efe7d326-5ac1-45b3-a6b1-ba350765852c-kube-api-access-99ls8\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.509510 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509462    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.509510 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.509695 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.509522    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-home\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.610599 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610545    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.610750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610622    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwpc\" (UniqueName: \"kubernetes.io/projected/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kube-api-access-5hwpc\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.610750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610644    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-dshm\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.610750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.610750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610693    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-model-cache\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.610750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.611031 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610776    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-home\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.611031 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ls8\" (UniqueName: \"kubernetes.io/projected/efe7d326-5ac1-45b3-a6b1-ba350765852c-kube-api-access-99ls8\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.611031 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.611031 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610894    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.611031 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610923    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-home\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.611031 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610962    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.611031 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.610986    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.611476 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.611062    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7d326-5ac1-45b3-a6b1-ba350765852c-tls-certs\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.611476 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.611157    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-model-cache\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.611476 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.611233    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.611476 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.611344    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-home\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.611720 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.611617    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-home\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.612026 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.611978    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.612026 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.611989    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.612595 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.612549    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.612760 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.612697    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.613338 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.613305    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-dshm\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.613454 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.613439    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.613777 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.613760    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7d326-5ac1-45b3-a6b1-ba350765852c-tls-certs\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.614160 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.614144    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.618746 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.618725    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwpc\" (UniqueName: \"kubernetes.io/projected/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kube-api-access-5hwpc\") pod \"router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.619304 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.619286    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ls8\" (UniqueName: \"kubernetes.io/projected/efe7d326-5ac1-45b3-a6b1-ba350765852c-kube-api-access-99ls8\") pod \"router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.650389 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.650349    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:45.676523 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.676497    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:45.678219 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.678200    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-q2c7x_5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:11:45.679647 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.679629    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:45.744215 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.744193    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-q2c7x_5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:11:45.744334 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.744233    2572 generic.go:356] "Generic (PLEG): container finished" podID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343" exitCode=137
Jul 20 11:11:45.744334 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.744288    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" event={"ID":"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a","Type":"ContainerDied","Data":"65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343"}
Jul 20 11:11:45.744334 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.744290    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x"
Jul 20 11:11:45.744442 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.744334    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-q2c7x" event={"ID":"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a","Type":"ContainerDied","Data":"294ed6f126cb37f93bb1d446cf233c8b0ff8825a4e3edc95a8834a8326ade9a8"}
Jul 20 11:11:45.744442 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.744362    2572 scope.go:117] "RemoveContainer" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343"
Jul 20 11:11:45.760895 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.760829    2572 scope.go:117] "RemoveContainer" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343"
Jul 20 11:11:45.761615 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:11:45.761572    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343\": container with ID starting with 65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343 not found: ID does not exist" containerID="65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343"
Jul 20 11:11:45.761742 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.761617    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343"} err="failed to get container status \"65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343\": rpc error: code = NotFound desc = could not find container \"65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343\": container with ID starting with 65f212130957008e1f0aa91cdc43309110dd6e9eac73273086d42412758ce343 not found: ID does not exist"
Jul 20 11:11:45.812807 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.812769    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-ready\") pod \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") "
Jul 20 11:11:45.813016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.812835    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-cni-sysctl-allowlist\") pod \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") "
Jul 20 11:11:45.813016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.812944    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twdjs\" (UniqueName: \"kubernetes.io/projected/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-kube-api-access-twdjs\") pod \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") "
Jul 20 11:11:45.813016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.812965    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"]
Jul 20 11:11:45.813016 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.812978    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-tuning-conf-dir\") pod \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\" (UID: \"5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a\") "
Jul 20 11:11:45.813267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.813115    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" (UID: "5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Jul 20 11:11:45.813267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.813145    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-ready" (OuterVolumeSpecName: "ready") pod "5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" (UID: "5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:45.813379 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.813213    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" (UID: "5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:11:45.813379 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.813305    2572 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-ready\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:45.813379 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.813349    2572 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-tuning-conf-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:45.815110 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:11:45.815083    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe7d326_5ac1_45b3_a6b1_ba350765852c.slice/crio-43352b9395220266bc5a1262d94eddf3f426495a1a5e9b85d6f199afffd645b7 WatchSource:0}: Error finding container 43352b9395220266bc5a1262d94eddf3f426495a1a5e9b85d6f199afffd645b7: Status 404 returned error can't find the container with id 43352b9395220266bc5a1262d94eddf3f426495a1a5e9b85d6f199afffd645b7
Jul 20 11:11:45.815284 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.815259    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-kube-api-access-twdjs" (OuterVolumeSpecName: "kube-api-access-twdjs") pod "5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" (UID: "5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a"). InnerVolumeSpecName "kube-api-access-twdjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:11:45.834623 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.834574    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"]
Jul 20 11:11:45.852726 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:11:45.852692    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9dfe3f6_a696_406d_9c7e_6d83cff829fe.slice/crio-f44eb4207c09d71ba20e436193b5b38a47c361db152a4845e232a6e9f8bbe7b6 WatchSource:0}: Error finding container f44eb4207c09d71ba20e436193b5b38a47c361db152a4845e232a6e9f8bbe7b6: Status 404 returned error can't find the container with id f44eb4207c09d71ba20e436193b5b38a47c361db152a4845e232a6e9f8bbe7b6
Jul 20 11:11:45.914157 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.914129    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twdjs\" (UniqueName: \"kubernetes.io/projected/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-kube-api-access-twdjs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:45.914157 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:45.914158    2572 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a-cni-sysctl-allowlist\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:11:46.066005 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:46.065962    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-q2c7x"]
Jul 20 11:11:46.069280 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:46.069250    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-q2c7x"]
Jul 20 11:11:46.519966 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:46.519930    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" path="/var/lib/kubelet/pods/5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a/volumes"
Jul 20 11:11:46.752357 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:46.752297    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" event={"ID":"efe7d326-5ac1-45b3-a6b1-ba350765852c","Type":"ContainerStarted","Data":"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216"}
Jul 20 11:11:46.752550 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:46.752386    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" event={"ID":"efe7d326-5ac1-45b3-a6b1-ba350765852c","Type":"ContainerStarted","Data":"43352b9395220266bc5a1262d94eddf3f426495a1a5e9b85d6f199afffd645b7"}
Jul 20 11:11:46.752550 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:46.752416    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:46.755478 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:46.755437    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" event={"ID":"f9dfe3f6-a696-406d-9c7e-6d83cff829fe","Type":"ContainerStarted","Data":"545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889"}
Jul 20 11:11:46.755687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:46.755500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" event={"ID":"f9dfe3f6-a696-406d-9c7e-6d83cff829fe","Type":"ContainerStarted","Data":"f44eb4207c09d71ba20e436193b5b38a47c361db152a4845e232a6e9f8bbe7b6"}
Jul 20 11:11:47.764195 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:47.764152    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" event={"ID":"efe7d326-5ac1-45b3-a6b1-ba350765852c","Type":"ContainerStarted","Data":"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64"}
Jul 20 11:11:49.782133 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:49.782090    2572 generic.go:356] "Generic (PLEG): container finished" podID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerID="545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889" exitCode=0
Jul 20 11:11:49.782593 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:49.782161    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" event={"ID":"f9dfe3f6-a696-406d-9c7e-6d83cff829fe","Type":"ContainerDied","Data":"545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889"}
Jul 20 11:11:50.788483 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:50.788439    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" event={"ID":"f9dfe3f6-a696-406d-9c7e-6d83cff829fe","Type":"ContainerStarted","Data":"a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297"}
Jul 20 11:11:50.790428 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:50.790398    2572 generic.go:356] "Generic (PLEG): container finished" podID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerID="cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64" exitCode=0
Jul 20 11:11:50.790612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:50.790425    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" event={"ID":"efe7d326-5ac1-45b3-a6b1-ba350765852c","Type":"ContainerDied","Data":"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64"}
Jul 20 11:11:50.811214 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:50.811154    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podStartSLOduration=2.171721091 podStartE2EDuration="5.811133666s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.639412575s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:11:50.809100749 +0000 UTC m=+4490.822202647" watchObservedRunningTime="2026-07-20 11:11:50.811133666 +0000 UTC m=+4490.824235563"
Jul 20 11:11:51.800342 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:51.800298    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" event={"ID":"efe7d326-5ac1-45b3-a6b1-ba350765852c","Type":"ContainerStarted","Data":"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49"}
Jul 20 11:11:51.829469 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:51.829411    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podStartSLOduration=3.344311083 podStartE2EDuration="6.829390347s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.485079264s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:11:51.826256038 +0000 UTC m=+4491.839357959" watchObservedRunningTime="2026-07-20 11:11:51.829390347 +0000 UTC m=+4491.842492309"
Jul 20 11:11:55.651403 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:55.651360    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:55.651897 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:55.651442    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:55.652924 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:55.652886    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:11:55.664909 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:55.664883    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:11:55.677246 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:55.677215    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:55.677533 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:55.677510    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:11:55.679031 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:11:55.679003    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:12:00.737871 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:00.737836    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:12:00.761095 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:00.761069    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:12:05.651406 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:05.651349    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:12:05.677643 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:05.677586    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:12:15.651516 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:15.651462    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:12:15.677042 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:15.676995    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:12:25.651129 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:25.651076    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:12:25.677675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:25.677624    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:12:35.651952 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:35.651899    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:12:35.676962 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:35.676913    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:12:45.650908 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:45.650858    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:12:45.677640 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:45.677584    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:12:55.651012 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:55.650961    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:12:55.676942 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:12:55.676893    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:13:05.651887 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:05.651840    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:13:05.677705 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:05.677662    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:13:15.651098 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:15.651040    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:13:15.677315 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:15.677266    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:13:25.651411 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:25.651355    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:13:25.677338 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:25.677299    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:13:35.651251 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:35.651184    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:13:35.677717 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:35.677674    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:13:45.651378 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:45.651317    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:13:45.677318 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:45.677270    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:13:55.651774 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:55.651722    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:13:55.677262 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:13:55.677218    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:14:05.651323 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:05.651273    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:14:05.677221 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:05.677172    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:14:15.651089 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:15.651033    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:14:15.676898 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:15.676852    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:14:25.650937 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:25.650884    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Jul 20 11:14:25.678816 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:25.678769    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 20 11:14:35.662725 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:35.662690    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:14:35.675489 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:35.675460    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:14:35.687522 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:35.687491    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:14:35.696136 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:14:35.696097    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:15:14.261734 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:14.261695    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"]
Jul 20 11:15:14.263361 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:14.263325    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"]
Jul 20 11:15:29.285766 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:29.285719    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main" containerID="cri-o://c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49" gracePeriod=45
Jul 20 11:15:29.286766 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:29.286706    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main" containerID="cri-o://a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297" gracePeriod=45
Jul 20 11:15:30.103697 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.103659    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="llm-d-routing-sidecar" containerID="cri-o://9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216" gracePeriod=45
Jul 20 11:15:30.246099 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.246072    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:15:30.289213 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289117    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-home\") pod \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") "
Jul 20 11:15:30.289213 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289179    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hwpc\" (UniqueName: \"kubernetes.io/projected/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kube-api-access-5hwpc\") pod \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") "
Jul 20 11:15:30.289213 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289205    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-model-cache\") pod \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") "
Jul 20 11:15:30.289803 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289287    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tmp-dir\") pod \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") "
Jul 20 11:15:30.289803 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289330    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tls-certs\") pod \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") "
Jul 20 11:15:30.289803 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289368    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-dshm\") pod \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") "
Jul 20 11:15:30.289803 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289395    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kserve-provision-location\") pod \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\" (UID: \"f9dfe3f6-a696-406d-9c7e-6d83cff829fe\") "
Jul 20 11:15:30.289803 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289597    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9dfe3f6-a696-406d-9c7e-6d83cff829fe" (UID: "f9dfe3f6-a696-406d-9c7e-6d83cff829fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.290084 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.289962    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-home" (OuterVolumeSpecName: "home") pod "f9dfe3f6-a696-406d-9c7e-6d83cff829fe" (UID: "f9dfe3f6-a696-406d-9c7e-6d83cff829fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.292670 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.292636    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-dshm" (OuterVolumeSpecName: "dshm") pod "f9dfe3f6-a696-406d-9c7e-6d83cff829fe" (UID: "f9dfe3f6-a696-406d-9c7e-6d83cff829fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.292796 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.292690    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kube-api-access-5hwpc" (OuterVolumeSpecName: "kube-api-access-5hwpc") pod "f9dfe3f6-a696-406d-9c7e-6d83cff829fe" (UID: "f9dfe3f6-a696-406d-9c7e-6d83cff829fe"). InnerVolumeSpecName "kube-api-access-5hwpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:15:30.292796 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.292721    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9dfe3f6-a696-406d-9c7e-6d83cff829fe" (UID: "f9dfe3f6-a696-406d-9c7e-6d83cff829fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:15:30.301990 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.301958    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9dfe3f6-a696-406d-9c7e-6d83cff829fe" (UID: "f9dfe3f6-a696-406d-9c7e-6d83cff829fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.356797 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.356759    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9dfe3f6-a696-406d-9c7e-6d83cff829fe" (UID: "f9dfe3f6-a696-406d-9c7e-6d83cff829fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.359191 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.359166    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:15:30.390006 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.389976    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-home\") pod \"efe7d326-5ac1-45b3-a6b1-ba350765852c\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") "
Jul 20 11:15:30.390175 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390013    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-dshm\") pod \"efe7d326-5ac1-45b3-a6b1-ba350765852c\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") "
Jul 20 11:15:30.390175 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390041    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7d326-5ac1-45b3-a6b1-ba350765852c-tls-certs\") pod \"efe7d326-5ac1-45b3-a6b1-ba350765852c\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") "
Jul 20 11:15:30.390175 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390064    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-kserve-provision-location\") pod \"efe7d326-5ac1-45b3-a6b1-ba350765852c\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") "
Jul 20 11:15:30.390175 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390097    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-model-cache\") pod \"efe7d326-5ac1-45b3-a6b1-ba350765852c\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") "
Jul 20 11:15:30.390411 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390178    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-tmp-dir\") pod \"efe7d326-5ac1-45b3-a6b1-ba350765852c\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") "
Jul 20 11:15:30.390411 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390202    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99ls8\" (UniqueName: \"kubernetes.io/projected/efe7d326-5ac1-45b3-a6b1-ba350765852c-kube-api-access-99ls8\") pod \"efe7d326-5ac1-45b3-a6b1-ba350765852c\" (UID: \"efe7d326-5ac1-45b3-a6b1-ba350765852c\") "
Jul 20 11:15:30.390527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390414    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.390527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390434    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.390527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390447    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.390527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390462    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwpc\" (UniqueName: \"kubernetes.io/projected/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-kube-api-access-5hwpc\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.390527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390475    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.390527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390488    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.390527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390501    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9dfe3f6-a696-406d-9c7e-6d83cff829fe-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.390931 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.390715    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-home" (OuterVolumeSpecName: "home") pod "efe7d326-5ac1-45b3-a6b1-ba350765852c" (UID: "efe7d326-5ac1-45b3-a6b1-ba350765852c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.391041 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.391019    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-model-cache" (OuterVolumeSpecName: "model-cache") pod "efe7d326-5ac1-45b3-a6b1-ba350765852c" (UID: "efe7d326-5ac1-45b3-a6b1-ba350765852c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.392441 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.392418    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe7d326-5ac1-45b3-a6b1-ba350765852c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efe7d326-5ac1-45b3-a6b1-ba350765852c" (UID: "efe7d326-5ac1-45b3-a6b1-ba350765852c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:15:30.392657 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.392640    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-dshm" (OuterVolumeSpecName: "dshm") pod "efe7d326-5ac1-45b3-a6b1-ba350765852c" (UID: "efe7d326-5ac1-45b3-a6b1-ba350765852c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.392941 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.392916    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe7d326-5ac1-45b3-a6b1-ba350765852c-kube-api-access-99ls8" (OuterVolumeSpecName: "kube-api-access-99ls8") pod "efe7d326-5ac1-45b3-a6b1-ba350765852c" (UID: "efe7d326-5ac1-45b3-a6b1-ba350765852c"). InnerVolumeSpecName "kube-api-access-99ls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:15:30.402716 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.402689    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efe7d326-5ac1-45b3-a6b1-ba350765852c" (UID: "efe7d326-5ac1-45b3-a6b1-ba350765852c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.452211 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.452169    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efe7d326-5ac1-45b3-a6b1-ba350765852c" (UID: "efe7d326-5ac1-45b3-a6b1-ba350765852c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:30.491891 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.491853    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.491891 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.491887    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99ls8\" (UniqueName: \"kubernetes.io/projected/efe7d326-5ac1-45b3-a6b1-ba350765852c-kube-api-access-99ls8\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.491891 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.491896    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.492169 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.491905    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.492169 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.491913    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7d326-5ac1-45b3-a6b1-ba350765852c-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.492169 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.491921    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.492169 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.491929    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7d326-5ac1-45b3-a6b1-ba350765852c-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:30.838991 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.838954    2572 generic.go:356] "Generic (PLEG): container finished" podID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerID="c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49" exitCode=0
Jul 20 11:15:30.838991 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.838984    2572 generic.go:356] "Generic (PLEG): container finished" podID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerID="9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216" exitCode=0
Jul 20 11:15:30.839280 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.839007    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"
Jul 20 11:15:30.839280 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.839039    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" event={"ID":"efe7d326-5ac1-45b3-a6b1-ba350765852c","Type":"ContainerDied","Data":"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49"}
Jul 20 11:15:30.839280 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.839093    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" event={"ID":"efe7d326-5ac1-45b3-a6b1-ba350765852c","Type":"ContainerDied","Data":"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216"}
Jul 20 11:15:30.839280 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.839114    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9" event={"ID":"efe7d326-5ac1-45b3-a6b1-ba350765852c","Type":"ContainerDied","Data":"43352b9395220266bc5a1262d94eddf3f426495a1a5e9b85d6f199afffd645b7"}
Jul 20 11:15:30.839280 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.839140    2572 scope.go:117] "RemoveContainer" containerID="c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49"
Jul 20 11:15:30.840731 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.840703    2572 generic.go:356] "Generic (PLEG): container finished" podID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerID="a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297" exitCode=0
Jul 20 11:15:30.840848 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.840741    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" event={"ID":"f9dfe3f6-a696-406d-9c7e-6d83cff829fe","Type":"ContainerDied","Data":"a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297"}
Jul 20 11:15:30.840848 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.840766    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"
Jul 20 11:15:30.840848 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.840778    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr" event={"ID":"f9dfe3f6-a696-406d-9c7e-6d83cff829fe","Type":"ContainerDied","Data":"f44eb4207c09d71ba20e436193b5b38a47c361db152a4845e232a6e9f8bbe7b6"}
Jul 20 11:15:30.847916 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.847895    2572 scope.go:117] "RemoveContainer" containerID="cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64"
Jul 20 11:15:30.855533 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.855516    2572 scope.go:117] "RemoveContainer" containerID="9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216"
Jul 20 11:15:30.858468 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.858446    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"]
Jul 20 11:15:30.863459 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.863440    2572 scope.go:117] "RemoveContainer" containerID="c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49"
Jul 20 11:15:30.863831 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:15:30.863812    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49\": container with ID starting with c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49 not found: ID does not exist" containerID="c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49"
Jul 20 11:15:30.863895 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.863841    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49"} err="failed to get container status \"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49\": rpc error: code = NotFound desc = could not find container \"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49\": container with ID starting with c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49 not found: ID does not exist"
Jul 20 11:15:30.863895 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.863861    2572 scope.go:117] "RemoveContainer" containerID="cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64"
Jul 20 11:15:30.863992 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.863919    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7dfcdcf8dd-vb2n9"]
Jul 20 11:15:30.864104 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:15:30.864084    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64\": container with ID starting with cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64 not found: ID does not exist" containerID="cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64"
Jul 20 11:15:30.864141 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.864110    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64"} err="failed to get container status \"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64\": rpc error: code = NotFound desc = could not find container \"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64\": container with ID starting with cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64 not found: ID does not exist"
Jul 20 11:15:30.864141 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.864125    2572 scope.go:117] "RemoveContainer" containerID="9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216"
Jul 20 11:15:30.864388 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:15:30.864369    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216\": container with ID starting with 9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216 not found: ID does not exist" containerID="9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216"
Jul 20 11:15:30.864440 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.864392    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216"} err="failed to get container status \"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216\": rpc error: code = NotFound desc = could not find container \"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216\": container with ID starting with 9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216 not found: ID does not exist"
Jul 20 11:15:30.864440 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.864406    2572 scope.go:117] "RemoveContainer" containerID="c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49"
Jul 20 11:15:30.864681 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.864661    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49"} err="failed to get container status \"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49\": rpc error: code = NotFound desc = could not find container \"c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49\": container with ID starting with c67f3b462fc381d5fbc9cfece05f8880feed7896ba32141e8ed99967cf2cab49 not found: ID does not exist"
Jul 20 11:15:30.864742 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.864681    2572 scope.go:117] "RemoveContainer" containerID="cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64"
Jul 20 11:15:30.864915 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.864882    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64"} err="failed to get container status \"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64\": rpc error: code = NotFound desc = could not find container \"cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64\": container with ID starting with cc98b72078b41db909fbea1c18de01ca3d1e9064d3853f707271fab15eb43d64 not found: ID does not exist"
Jul 20 11:15:30.864967 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.864916    2572 scope.go:117] "RemoveContainer" containerID="9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216"
Jul 20 11:15:30.865147 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.865121    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216"} err="failed to get container status \"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216\": rpc error: code = NotFound desc = could not find container \"9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216\": container with ID starting with 9378f8510f04556a4150fa94ed15be347f6e43f7c2ba08436988245f5dcd2216 not found: ID does not exist"
Jul 20 11:15:30.865147 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.865145    2572 scope.go:117] "RemoveContainer" containerID="a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297"
Jul 20 11:15:30.872821 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.872798    2572 scope.go:117] "RemoveContainer" containerID="545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889"
Jul 20 11:15:30.874353 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.874331    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"]
Jul 20 11:15:30.878246 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.878223    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-c68f976f5-69qgr"]
Jul 20 11:15:30.881060 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.881044    2572 scope.go:117] "RemoveContainer" containerID="a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297"
Jul 20 11:15:30.881302 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:15:30.881281    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297\": container with ID starting with a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297 not found: ID does not exist" containerID="a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297"
Jul 20 11:15:30.881372 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.881310    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297"} err="failed to get container status \"a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297\": rpc error: code = NotFound desc = could not find container \"a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297\": container with ID starting with a4bdcac13376b9944b52ae06958fbc532d58b3048ac54544b0cc960a54c1f297 not found: ID does not exist"
Jul 20 11:15:30.881372 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.881333    2572 scope.go:117] "RemoveContainer" containerID="545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889"
Jul 20 11:15:30.881606 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:15:30.881588    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889\": container with ID starting with 545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889 not found: ID does not exist" containerID="545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889"
Jul 20 11:15:30.881652 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:30.881611    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889"} err="failed to get container status \"545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889\": rpc error: code = NotFound desc = could not find container \"545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889\": container with ID starting with 545b30ec5836de1e8665f0b7d2e85fbeb8402e348943fa82b323f2de6e685889 not found: ID does not exist"
Jul 20 11:15:32.519526 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:32.519482    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" path="/var/lib/kubelet/pods/efe7d326-5ac1-45b3-a6b1-ba350765852c/volumes"
Jul 20 11:15:32.519994 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:32.519979    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" path="/var/lib/kubelet/pods/f9dfe3f6-a696-406d-9c7e-6d83cff829fe/volumes"
Jul 20 11:15:39.208833 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:39.208792    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"]
Jul 20 11:15:39.209629 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:39.209015    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy" containerID="cri-o://cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10" gracePeriod=30
Jul 20 11:15:44.454333 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.454303    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:15:44.506875 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.506847    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-podinfo\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507053 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.506893    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-data\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507053 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.506931    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2c5b1-d360-4c03-93e5-654c21ff2082-istiod-ca-cert\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507053 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.506991    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-certs\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507053 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507047    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-envoy\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507272 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507071    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-socket\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507272 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507111    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96hmp\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-kube-api-access-96hmp\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507272 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507135    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-token\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507272 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507165    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-credential-socket\") pod \"41f2c5b1-d360-4c03-93e5-654c21ff2082\" (UID: \"41f2c5b1-d360-4c03-93e5-654c21ff2082\") "
Jul 20 11:15:44.507475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507251    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:44.507475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507278    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-data" (OuterVolumeSpecName: "istio-data") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:44.507475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507381    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f2c5b1-d360-4c03-93e5-654c21ff2082-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:15:44.507475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507441    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.507475 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507463    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-data\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.507701 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507497    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:44.507701 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.507514    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:44.509675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.509630    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 11:15:44.509675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.509641    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-kube-api-access-96hmp" (OuterVolumeSpecName: "kube-api-access-96hmp") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "kube-api-access-96hmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:15:44.509895 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.509782    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:44.509964 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.509910    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-token" (OuterVolumeSpecName: "istio-token") pod "41f2c5b1-d360-4c03-93e5-654c21ff2082" (UID: "41f2c5b1-d360-4c03-93e5-654c21ff2082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:15:44.608577 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.608509    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-envoy\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.608577 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.608548    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-workload-socket\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.608855 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.608586    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96hmp\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-kube-api-access-96hmp\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.608855 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.608601    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-token\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.608855 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.608614    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2c5b1-d360-4c03-93e5-654c21ff2082-credential-socket\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.608855 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.608625    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2c5b1-d360-4c03-93e5-654c21ff2082-istio-podinfo\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.608855 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.608639    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2c5b1-d360-4c03-93e5-654c21ff2082-istiod-ca-cert\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:15:44.901620 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.901501    2572 generic.go:356] "Generic (PLEG): container finished" podID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerID="cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10" exitCode=0
Jul 20 11:15:44.901620 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.901545    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"
Jul 20 11:15:44.901620 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.901598    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" event={"ID":"41f2c5b1-d360-4c03-93e5-654c21ff2082","Type":"ContainerDied","Data":"cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10"}
Jul 20 11:15:44.901846 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.901655    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv" event={"ID":"41f2c5b1-d360-4c03-93e5-654c21ff2082","Type":"ContainerDied","Data":"5f92e09574120fbde0d06d8f3fd81d436c8eb936fa515f08b86c9fd339c744c6"}
Jul 20 11:15:44.901846 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.901677    2572 scope.go:117] "RemoveContainer" containerID="cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10"
Jul 20 11:15:44.910021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.910005    2572 scope.go:117] "RemoveContainer" containerID="cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7"
Jul 20 11:15:44.917726 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.917665    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"]
Jul 20 11:15:44.917805 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.917740    2572 scope.go:117] "RemoveContainer" containerID="cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10"
Jul 20 11:15:44.918011 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:15:44.917993    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10\": container with ID starting with cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10 not found: ID does not exist" containerID="cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10"
Jul 20 11:15:44.918079 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.918023    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10"} err="failed to get container status \"cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10\": rpc error: code = NotFound desc = could not find container \"cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10\": container with ID starting with cf70d19fba49a78a52c5508b8285d09c87239f9057ff5320d009d951777f1d10 not found: ID does not exist"
Jul 20 11:15:44.918079 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.918048    2572 scope.go:117] "RemoveContainer" containerID="cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7"
Jul 20 11:15:44.918305 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:15:44.918287    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7\": container with ID starting with cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7 not found: ID does not exist" containerID="cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7"
Jul 20 11:15:44.918397 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.918313    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7"} err="failed to get container status \"cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7\": rpc error: code = NotFound desc = could not find container \"cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7\": container with ID starting with cf959e96e5161248c0b630bb952c90a2452108b75c9b4e25ab845ac8750433e7 not found: ID does not exist"
Jul 20 11:15:44.921316 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:44.921293    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-66786c46c-tpkdv"]
Jul 20 11:15:45.398748 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.398716    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"]
Jul 20 11:15:45.399012 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399001    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main"
Jul 20 11:15:45.399061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399014    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main"
Jul 20 11:15:45.399061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399024    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="storage-initializer"
Jul 20 11:15:45.399061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399030    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="storage-initializer"
Jul 20 11:15:45.399061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399038    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="llm-d-routing-sidecar"
Jul 20 11:15:45.399061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399045    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="llm-d-routing-sidecar"
Jul 20 11:15:45.399061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399053    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main"
Jul 20 11:15:45.399061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399059    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399065    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399071    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399083    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399087    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399093    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="storage-initializer"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399098    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="storage-initializer"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399105    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399110    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399158    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="main"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399165    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9dfe3f6-a696-406d-9c7e-6d83cff829fe" containerName="main"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399172    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399179    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5058bd63-cc3b-45ea-bf1d-30b5b29b0d1a" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399186    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe7d326-5ac1-45b3-a6b1-ba350765852c" containerName="llm-d-routing-sidecar"
Jul 20 11:15:45.399270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.399273    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" containerName="istio-proxy"
Jul 20 11:15:45.403669 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.403651    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.406298 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.406268    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-nwknn\""
Jul 20 11:15:45.406431 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.406320    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 20 11:15:45.406431 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.406334    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 20 11:15:45.406431 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.406331    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 20 11:15:45.406431 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.406330    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 20 11:15:45.411871 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.411846    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"]
Jul 20 11:15:45.515692 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.515655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75926cb8-7b62-46cc-9156-b8397976017f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.516061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.515699    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.516061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.515727    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.516061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.515744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.516061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.515762    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.516061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.515784    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4vn\" (UniqueName: \"kubernetes.io/projected/75926cb8-7b62-46cc-9156-b8397976017f-kube-api-access-8k4vn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.516061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.515873    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.616633 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.616589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.616836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.616646    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75926cb8-7b62-46cc-9156-b8397976017f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.616836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.616681    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.616836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.616704    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.616836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.616723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.616836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.616751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.617107 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.616834    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4vn\" (UniqueName: \"kubernetes.io/projected/75926cb8-7b62-46cc-9156-b8397976017f-kube-api-access-8k4vn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.617107 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.617081    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.617250 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.617108    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.617250 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.617166    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.617250 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.617225    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.619091 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.619065    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.619449 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.619429    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75926cb8-7b62-46cc-9156-b8397976017f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.624825 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.624802    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4vn\" (UniqueName: \"kubernetes.io/projected/75926cb8-7b62-46cc-9156-b8397976017f-kube-api-access-8k4vn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.714149 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.714116    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:45.884293 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.884256    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"]
Jul 20 11:15:45.888589 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:15:45.888541    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75926cb8_7b62_46cc_9156_b8397976017f.slice/crio-e2cc27faf605ed13bf2b5646f718ee0454e0a2210b3bdab16c41df5e00543109 WatchSource:0}: Error finding container e2cc27faf605ed13bf2b5646f718ee0454e0a2210b3bdab16c41df5e00543109: Status 404 returned error can't find the container with id e2cc27faf605ed13bf2b5646f718ee0454e0a2210b3bdab16c41df5e00543109
Jul 20 11:15:45.907706 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:45.907669    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" event={"ID":"75926cb8-7b62-46cc-9156-b8397976017f","Type":"ContainerStarted","Data":"e2cc27faf605ed13bf2b5646f718ee0454e0a2210b3bdab16c41df5e00543109"}
Jul 20 11:15:46.519541 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:46.519505    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f2c5b1-d360-4c03-93e5-654c21ff2082" path="/var/lib/kubelet/pods/41f2c5b1-d360-4c03-93e5-654c21ff2082/volumes"
Jul 20 11:15:46.917519 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:46.917429    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" event={"ID":"75926cb8-7b62-46cc-9156-b8397976017f","Type":"ContainerStarted","Data":"9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073"}
Jul 20 11:15:49.934969 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:49.934930    2572 generic.go:356] "Generic (PLEG): container finished" podID="75926cb8-7b62-46cc-9156-b8397976017f" containerID="9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073" exitCode=0
Jul 20 11:15:49.935470 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:49.934995    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" event={"ID":"75926cb8-7b62-46cc-9156-b8397976017f","Type":"ContainerDied","Data":"9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073"}
Jul 20 11:15:50.941837 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:50.941796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" event={"ID":"75926cb8-7b62-46cc-9156-b8397976017f","Type":"ContainerStarted","Data":"b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda"}
Jul 20 11:15:50.964209 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:50.964145    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podStartSLOduration=2.479316788 podStartE2EDuration="5.964122511s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484805723s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:15:50.960829478 +0000 UTC m=+4730.973931385" watchObservedRunningTime="2026-07-20 11:15:50.964122511 +0000 UTC m=+4730.977224410"
Jul 20 11:15:51.714891 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:51.714859    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:55.715300 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:55.715258    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:15:55.716635 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:15:55.716600    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:16:05.715011 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:16:05.714961    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:16:15.714678 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:16:15.714627    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:16:25.715341 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:16:25.715292    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:16:35.715457 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:16:35.715403    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:16:45.715012 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:16:45.714961    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:16:55.715421 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:16:55.715368    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:17:00.763029 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:00.763002    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:17:00.793292 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:00.793257    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:17:05.715119 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:05.715072    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:17:15.714655 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:15.714607    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:17:25.715018 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:25.714973    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:17:35.715124 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:35.715070    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:17:45.724744 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:45.724706    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:17:45.733237 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:45.733213    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:17:50.445879 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:17:50.445846    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"]
Jul 20 11:18:05.469324 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:05.469277    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" containerID="cri-o://b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda" gracePeriod=45
Jul 20 11:18:05.725500 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:05.725404    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 20 11:18:06.218593 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.218569    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:18:06.395787 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.395695    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-tmp-dir\") pod \"75926cb8-7b62-46cc-9156-b8397976017f\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") "
Jul 20 11:18:06.395787 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.395745    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-dshm\") pod \"75926cb8-7b62-46cc-9156-b8397976017f\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") "
Jul 20 11:18:06.395787 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.395762    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-model-cache\") pod \"75926cb8-7b62-46cc-9156-b8397976017f\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") "
Jul 20 11:18:06.395787 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.395779    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-kserve-provision-location\") pod \"75926cb8-7b62-46cc-9156-b8397976017f\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") "
Jul 20 11:18:06.396199 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.395811    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75926cb8-7b62-46cc-9156-b8397976017f-tls-certs\") pod \"75926cb8-7b62-46cc-9156-b8397976017f\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") "
Jul 20 11:18:06.396199 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.395831    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-home\") pod \"75926cb8-7b62-46cc-9156-b8397976017f\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") "
Jul 20 11:18:06.396199 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.395873    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k4vn\" (UniqueName: \"kubernetes.io/projected/75926cb8-7b62-46cc-9156-b8397976017f-kube-api-access-8k4vn\") pod \"75926cb8-7b62-46cc-9156-b8397976017f\" (UID: \"75926cb8-7b62-46cc-9156-b8397976017f\") "
Jul 20 11:18:06.396199 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.396050    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-model-cache" (OuterVolumeSpecName: "model-cache") pod "75926cb8-7b62-46cc-9156-b8397976017f" (UID: "75926cb8-7b62-46cc-9156-b8397976017f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:18:06.396544 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.396517    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-home" (OuterVolumeSpecName: "home") pod "75926cb8-7b62-46cc-9156-b8397976017f" (UID: "75926cb8-7b62-46cc-9156-b8397976017f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:18:06.398061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.398030    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75926cb8-7b62-46cc-9156-b8397976017f-kube-api-access-8k4vn" (OuterVolumeSpecName: "kube-api-access-8k4vn") pod "75926cb8-7b62-46cc-9156-b8397976017f" (UID: "75926cb8-7b62-46cc-9156-b8397976017f"). InnerVolumeSpecName "kube-api-access-8k4vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:18:06.398061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.398048    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75926cb8-7b62-46cc-9156-b8397976017f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75926cb8-7b62-46cc-9156-b8397976017f" (UID: "75926cb8-7b62-46cc-9156-b8397976017f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:18:06.398384 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.398371    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-dshm" (OuterVolumeSpecName: "dshm") pod "75926cb8-7b62-46cc-9156-b8397976017f" (UID: "75926cb8-7b62-46cc-9156-b8397976017f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:18:06.408616 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.408591    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75926cb8-7b62-46cc-9156-b8397976017f" (UID: "75926cb8-7b62-46cc-9156-b8397976017f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:18:06.466508 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.466466    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75926cb8-7b62-46cc-9156-b8397976017f" (UID: "75926cb8-7b62-46cc-9156-b8397976017f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:18:06.496968 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.496943    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k4vn\" (UniqueName: \"kubernetes.io/projected/75926cb8-7b62-46cc-9156-b8397976017f-kube-api-access-8k4vn\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:18:06.496968 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.496966    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:18:06.496968 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.496976    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:18:06.497502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.496984    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:18:06.497502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.496991    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:18:06.497502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.497001    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75926cb8-7b62-46cc-9156-b8397976017f-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:18:06.497502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.497010    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75926cb8-7b62-46cc-9156-b8397976017f-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:18:06.500837 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.500807    2572 generic.go:356] "Generic (PLEG): container finished" podID="75926cb8-7b62-46cc-9156-b8397976017f" containerID="b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda" exitCode=0
Jul 20 11:18:06.500968 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.500872    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"
Jul 20 11:18:06.500968 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.500893    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" event={"ID":"75926cb8-7b62-46cc-9156-b8397976017f","Type":"ContainerDied","Data":"b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda"}
Jul 20 11:18:06.500968 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.500951    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g" event={"ID":"75926cb8-7b62-46cc-9156-b8397976017f","Type":"ContainerDied","Data":"e2cc27faf605ed13bf2b5646f718ee0454e0a2210b3bdab16c41df5e00543109"}
Jul 20 11:18:06.501142 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.500975    2572 scope.go:117] "RemoveContainer" containerID="b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda"
Jul 20 11:18:06.512005 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.511979    2572 scope.go:117] "RemoveContainer" containerID="9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073"
Jul 20 11:18:06.520046 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.520025    2572 scope.go:117] "RemoveContainer" containerID="b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda"
Jul 20 11:18:06.520311 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:18:06.520287    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda\": container with ID starting with b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda not found: ID does not exist" containerID="b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda"
Jul 20 11:18:06.520368 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.520320    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda"} err="failed to get container status \"b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda\": rpc error: code = NotFound desc = could not find container \"b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda\": container with ID starting with b7949ba7f54e7ae30cb23fa52ea7dc6cb8a1710f2dd45a717837459c70272fda not found: ID does not exist"
Jul 20 11:18:06.520368 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.520338    2572 scope.go:117] "RemoveContainer" containerID="9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073"
Jul 20 11:18:06.520567 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:18:06.520540    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073\": container with ID starting with 9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073 not found: ID does not exist" containerID="9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073"
Jul 20 11:18:06.520612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.520578    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073"} err="failed to get container status \"9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073\": rpc error: code = NotFound desc = could not find container \"9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073\": container with ID starting with 9f37ed971a33d8e7c4cf39b6b4bb7d42474d6762442482a725001ed4602fe073 not found: ID does not exist"
Jul 20 11:18:06.531368 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.531341    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"]
Jul 20 11:18:06.540360 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:06.540337    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-8dd75c4cc-qw82g"]
Jul 20 11:18:08.518621 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:08.518588    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75926cb8-7b62-46cc-9156-b8397976017f" path="/var/lib/kubelet/pods/75926cb8-7b62-46cc-9156-b8397976017f/volumes"
Jul 20 11:18:20.076625 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.076587    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 20 11:18:20.077142 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.076878    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="storage-initializer"
Jul 20 11:18:20.077142 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.076889    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="storage-initializer"
Jul 20 11:18:20.077142 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.076902    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main"
Jul 20 11:18:20.077142 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.076907    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main"
Jul 20 11:18:20.077142 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.076957    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="75926cb8-7b62-46cc-9156-b8397976017f" containerName="main"
Jul 20 11:18:20.081700 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.081678    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.084438 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.084412    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 20 11:18:20.085544 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.085518    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 20 11:18:20.085681 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.085543    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 20 11:18:20.085681 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.085612    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-q5dn8\""
Jul 20 11:18:20.085681 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.085549    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 20 11:18:20.085681 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.085520    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-q9mxj\""
Jul 20 11:18:20.087479 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.087455    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.087587 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.087510    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.087645 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.087592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.087645 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.087625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vxh\" (UniqueName: \"kubernetes.io/projected/ee80a3ca-a152-4f4b-b440-015d9f01150a-kube-api-access-64vxh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.087717 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.087667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee80a3ca-a152-4f4b-b440-015d9f01150a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.087717 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.087692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.087786 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.087767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.094203 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.094174    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 20 11:18:20.173527 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.173491    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 20 11:18:20.176767 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.176750    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.188797 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.188770    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.188937 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.188807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.188937 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.188825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.189035 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189008    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.189097 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189044    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.189097 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189073    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.189097 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189079    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.189267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189109    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vxh\" (UniqueName: \"kubernetes.io/projected/ee80a3ca-a152-4f4b-b440-015d9f01150a-kube-api-access-64vxh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.189267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189137    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.189267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189168    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbqq\" (UniqueName: \"kubernetes.io/projected/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kube-api-access-jkbqq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.189267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189202    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.189267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189228    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee80a3ca-a152-4f4b-b440-015d9f01150a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.189267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189231    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.189267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189253    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.189577 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189298    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.189577 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189326    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.189577 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189328    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.189577 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.189535    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.191055 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.191039    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.191445 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.191430    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee80a3ca-a152-4f4b-b440-015d9f01150a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.200174 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.200146    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 20 11:18:20.209722 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.209698    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vxh\" (UniqueName: \"kubernetes.io/projected/ee80a3ca-a152-4f4b-b440-015d9f01150a-kube-api-access-64vxh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.290541 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.290496    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.290750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.290548    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.290750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.290588    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbqq\" (UniqueName: \"kubernetes.io/projected/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kube-api-access-jkbqq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.290750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.290612    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.290750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.290629    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.290750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.290644    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.290750 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.290699    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.291068 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.290941    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.291129 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.291086    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.291129 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.291101    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.291129 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.291122    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.292990 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.292969    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.293162 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.293137    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.298868 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.298847    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbqq\" (UniqueName: \"kubernetes.io/projected/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kube-api-access-jkbqq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.391510 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.391429    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:20.486679 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.486647    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:18:20.543481 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.543448    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 20 11:18:20.545695 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:18:20.545667    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee80a3ca_a152_4f4b_b440_015d9f01150a.slice/crio-37ba83aceabef1c6cfd45f16b56b6a37bd83c15fe30096f2346515122e170bc2 WatchSource:0}: Error finding container 37ba83aceabef1c6cfd45f16b56b6a37bd83c15fe30096f2346515122e170bc2: Status 404 returned error can't find the container with id 37ba83aceabef1c6cfd45f16b56b6a37bd83c15fe30096f2346515122e170bc2
Jul 20 11:18:20.557835 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.557807    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ee80a3ca-a152-4f4b-b440-015d9f01150a","Type":"ContainerStarted","Data":"37ba83aceabef1c6cfd45f16b56b6a37bd83c15fe30096f2346515122e170bc2"}
Jul 20 11:18:20.627987 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:20.627947    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 20 11:18:20.631950 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:18:20.631903    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041be8f8_2cf7_44e3_9b50_883dd8b26a7f.slice/crio-ef36776e0097685fd0506d7252a397e7bd539f5482669a6bd0b187c923514c15 WatchSource:0}: Error finding container ef36776e0097685fd0506d7252a397e7bd539f5482669a6bd0b187c923514c15: Status 404 returned error can't find the container with id ef36776e0097685fd0506d7252a397e7bd539f5482669a6bd0b187c923514c15
Jul 20 11:18:21.567785 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:21.567735    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"041be8f8-2cf7-44e3-9b50-883dd8b26a7f","Type":"ContainerStarted","Data":"835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8"}
Jul 20 11:18:21.568330 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:21.567869    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"041be8f8-2cf7-44e3-9b50-883dd8b26a7f","Type":"ContainerStarted","Data":"ef36776e0097685fd0506d7252a397e7bd539f5482669a6bd0b187c923514c15"}
Jul 20 11:18:21.569618 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:21.569584    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ee80a3ca-a152-4f4b-b440-015d9f01150a","Type":"ContainerStarted","Data":"11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7"}
Jul 20 11:18:24.585104 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:24.585069    2572 generic.go:356] "Generic (PLEG): container finished" podID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerID="11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7" exitCode=0
Jul 20 11:18:24.585581 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:24.585148    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ee80a3ca-a152-4f4b-b440-015d9f01150a","Type":"ContainerDied","Data":"11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7"}
Jul 20 11:18:24.586724 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:24.586697    2572 generic.go:356] "Generic (PLEG): container finished" podID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" containerID="835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8" exitCode=0
Jul 20 11:18:24.586813 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:24.586767    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"041be8f8-2cf7-44e3-9b50-883dd8b26a7f","Type":"ContainerDied","Data":"835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8"}
Jul 20 11:18:25.594795 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:25.594753    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"041be8f8-2cf7-44e3-9b50-883dd8b26a7f","Type":"ContainerStarted","Data":"0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813"}
Jul 20 11:18:25.596795 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:25.596770    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ee80a3ca-a152-4f4b-b440-015d9f01150a","Type":"ContainerStarted","Data":"b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab"}
Jul 20 11:18:25.618987 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:25.618918    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=1.9374847960000001 podStartE2EDuration="5.618892074s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.681407278s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:18:25.617390839 +0000 UTC m=+4885.630492737" watchObservedRunningTime="2026-07-20 11:18:25.618892074 +0000 UTC m=+4885.631993981"
Jul 20 11:18:25.638896 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:25.638836    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=1.769692337 podStartE2EDuration="5.638819354s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.869127017s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:18:25.637601679 +0000 UTC m=+4885.650703576" watchObservedRunningTime="2026-07-20 11:18:25.638819354 +0000 UTC m=+4885.651921249"
Jul 20 11:18:26.392014 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:26.391965    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:30.392032 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:30.391984    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:18:30.393857 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:30.393823    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:18:40.392883 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:40.392822    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:18:50.392170 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:18:50.392118    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:19:00.392289 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:19:00.392242    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:19:10.391980 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:19:10.391924    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:19:20.392667 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:19:20.392618    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:19:30.392719 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:19:30.392611    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:19:40.392706 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:19:40.392654    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:19:50.392261 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:19:50.392213    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:20:00.392331 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:00.392284    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:20:10.392736 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:10.392676    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:20:20.391991 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:20.391936    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:20:30.392380 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:30.392320    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:20:40.392452 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:40.392396    2572 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Jul 20 11:20:50.404481 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:50.404443    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:20:50.412045 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:50.412018    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:20:55.407670 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:55.407579    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 20 11:20:55.514381 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:20:55.514347    2572 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-q5dn8\" not found"
Jul 20 11:20:55.563594 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:20:55.562431    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:20:55.563594 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:20:55.562540    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs podName:041be8f8-2cf7-44e3-9b50-883dd8b26a7f nodeName:}" failed. No retries permitted until 2026-07-20 11:20:56.062516549 +0000 UTC m=+5036.075618426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:20:56.066404 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:20:56.066367    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:20:56.066622 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:20:56.066440    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs podName:041be8f8-2cf7-44e3-9b50-883dd8b26a7f nodeName:}" failed. No retries permitted until 2026-07-20 11:20:57.066425615 +0000 UTC m=+5037.079527488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:20:57.074649 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:20:57.074608    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:20:57.075073 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:20:57.074695    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs podName:041be8f8-2cf7-44e3-9b50-883dd8b26a7f nodeName:}" failed. No retries permitted until 2026-07-20 11:20:59.07467838 +0000 UTC m=+5039.087780254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:20:59.091801 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:20:59.091760    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:20:59.092225 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:20:59.091841    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs podName:041be8f8-2cf7-44e3-9b50-883dd8b26a7f nodeName:}" failed. No retries permitted until 2026-07-20 11:21:03.091821932 +0000 UTC m=+5043.104923823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:21:03.126067 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:03.126019    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:21:03.126496 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:03.126125    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs podName:041be8f8-2cf7-44e3-9b50-883dd8b26a7f nodeName:}" failed. No retries permitted until 2026-07-20 11:21:11.126108027 +0000 UTC m=+5051.139209902 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:21:10.431268 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:10.431198    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main" containerID="cri-o://b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab" gracePeriod=45
Jul 20 11:21:11.195407 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.195380    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:21:11.200154 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:11.200129    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:21:11.200289 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:11.200190    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs podName:041be8f8-2cf7-44e3-9b50-883dd8b26a7f nodeName:}" failed. No retries permitted until 2026-07-20 11:21:27.200175511 +0000 UTC m=+5067.213277385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:21:11.301352 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301254    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee80a3ca-a152-4f4b-b440-015d9f01150a-tls-certs\") pod \"ee80a3ca-a152-4f4b-b440-015d9f01150a\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") "
Jul 20 11:21:11.301352 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301308    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-model-cache\") pod \"ee80a3ca-a152-4f4b-b440-015d9f01150a\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") "
Jul 20 11:21:11.301612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301350    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-kserve-provision-location\") pod \"ee80a3ca-a152-4f4b-b440-015d9f01150a\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") "
Jul 20 11:21:11.301612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301377    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-dshm\") pod \"ee80a3ca-a152-4f4b-b440-015d9f01150a\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") "
Jul 20 11:21:11.301612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301401    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-home\") pod \"ee80a3ca-a152-4f4b-b440-015d9f01150a\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") "
Jul 20 11:21:11.301612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301423    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64vxh\" (UniqueName: \"kubernetes.io/projected/ee80a3ca-a152-4f4b-b440-015d9f01150a-kube-api-access-64vxh\") pod \"ee80a3ca-a152-4f4b-b440-015d9f01150a\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") "
Jul 20 11:21:11.301612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301446    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-tmp-dir\") pod \"ee80a3ca-a152-4f4b-b440-015d9f01150a\" (UID: \"ee80a3ca-a152-4f4b-b440-015d9f01150a\") "
Jul 20 11:21:11.301887 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301618    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee80a3ca-a152-4f4b-b440-015d9f01150a" (UID: "ee80a3ca-a152-4f4b-b440-015d9f01150a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:11.301887 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.301701    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:11.302277 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.302240    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-home" (OuterVolumeSpecName: "home") pod "ee80a3ca-a152-4f4b-b440-015d9f01150a" (UID: "ee80a3ca-a152-4f4b-b440-015d9f01150a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:11.303902 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.303873    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee80a3ca-a152-4f4b-b440-015d9f01150a-kube-api-access-64vxh" (OuterVolumeSpecName: "kube-api-access-64vxh") pod "ee80a3ca-a152-4f4b-b440-015d9f01150a" (UID: "ee80a3ca-a152-4f4b-b440-015d9f01150a"). InnerVolumeSpecName "kube-api-access-64vxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:21:11.304038 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.304021    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee80a3ca-a152-4f4b-b440-015d9f01150a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee80a3ca-a152-4f4b-b440-015d9f01150a" (UID: "ee80a3ca-a152-4f4b-b440-015d9f01150a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:21:11.304409 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.304394    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-dshm" (OuterVolumeSpecName: "dshm") pod "ee80a3ca-a152-4f4b-b440-015d9f01150a" (UID: "ee80a3ca-a152-4f4b-b440-015d9f01150a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:11.315244 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.315212    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee80a3ca-a152-4f4b-b440-015d9f01150a" (UID: "ee80a3ca-a152-4f4b-b440-015d9f01150a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:11.355886 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.355840    2572 generic.go:356] "Generic (PLEG): container finished" podID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerID="b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab" exitCode=0
Jul 20 11:21:11.356051 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.355905    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 20 11:21:11.356051 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.355920    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ee80a3ca-a152-4f4b-b440-015d9f01150a","Type":"ContainerDied","Data":"b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab"}
Jul 20 11:21:11.356051 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.355984    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ee80a3ca-a152-4f4b-b440-015d9f01150a","Type":"ContainerDied","Data":"37ba83aceabef1c6cfd45f16b56b6a37bd83c15fe30096f2346515122e170bc2"}
Jul 20 11:21:11.356051 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.356009    2572 scope.go:117] "RemoveContainer" containerID="b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab"
Jul 20 11:21:11.364638 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.364606    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee80a3ca-a152-4f4b-b440-015d9f01150a" (UID: "ee80a3ca-a152-4f4b-b440-015d9f01150a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:11.365686 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.365670    2572 scope.go:117] "RemoveContainer" containerID="11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7"
Jul 20 11:21:11.373727 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.373706    2572 scope.go:117] "RemoveContainer" containerID="b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab"
Jul 20 11:21:11.373996 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:11.373977    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab\": container with ID starting with b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab not found: ID does not exist" containerID="b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab"
Jul 20 11:21:11.374043 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.374007    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab"} err="failed to get container status \"b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab\": rpc error: code = NotFound desc = could not find container \"b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab\": container with ID starting with b2e1b32b8ed7c9146ee57138027ccdbe1d9e10da8f19c9626e5b139e23e01aab not found: ID does not exist"
Jul 20 11:21:11.374043 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.374027    2572 scope.go:117] "RemoveContainer" containerID="11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7"
Jul 20 11:21:11.374251 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:11.374233    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7\": container with ID starting with 11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7 not found: ID does not exist" containerID="11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7"
Jul 20 11:21:11.374295 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.374255    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7"} err="failed to get container status \"11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7\": rpc error: code = NotFound desc = could not find container \"11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7\": container with ID starting with 11f0a99ce151c7493ed25baa333d147fed554e028c7dff4375ce03c4b9e00ce7 not found: ID does not exist"
Jul 20 11:21:11.402178 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.402149    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:11.402178 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.402177    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:11.402344 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.402187    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:11.402344 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.402196    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vxh\" (UniqueName: \"kubernetes.io/projected/ee80a3ca-a152-4f4b-b440-015d9f01150a-kube-api-access-64vxh\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:11.402344 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.402205    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee80a3ca-a152-4f4b-b440-015d9f01150a-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:11.402344 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.402213    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee80a3ca-a152-4f4b-b440-015d9f01150a-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:11.677417 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.677387    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 20 11:21:11.682300 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.682275    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 20 11:21:11.696602 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:11.696571    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 20 11:21:12.519716 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:12.519682    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" path="/var/lib/kubelet/pods/ee80a3ca-a152-4f4b-b440-015d9f01150a/volumes"
Jul 20 11:21:26.722814 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:26.722738    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" containerName="main" containerID="cri-o://0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813" gracePeriod=45
Jul 20 11:21:27.251518 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:27.251486    2572 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:21:27.251687 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:27.251593    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs podName:041be8f8-2cf7-44e3-9b50-883dd8b26a7f nodeName:}" failed. No retries permitted until 2026-07-20 11:21:59.251549345 +0000 UTC m=+5099.264651220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 20 11:21:27.395799 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.395772    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:21:27.433297 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.433258    2572 generic.go:356] "Generic (PLEG): container finished" podID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" containerID="0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813" exitCode=0
Jul 20 11:21:27.433467 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.433311    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"041be8f8-2cf7-44e3-9b50-883dd8b26a7f","Type":"ContainerDied","Data":"0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813"}
Jul 20 11:21:27.433467 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.433332    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 20 11:21:27.433467 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.433354    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"041be8f8-2cf7-44e3-9b50-883dd8b26a7f","Type":"ContainerDied","Data":"ef36776e0097685fd0506d7252a397e7bd539f5482669a6bd0b187c923514c15"}
Jul 20 11:21:27.433467 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.433375    2572 scope.go:117] "RemoveContainer" containerID="0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813"
Jul 20 11:21:27.441820 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.441799    2572 scope.go:117] "RemoveContainer" containerID="835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8"
Jul 20 11:21:27.449629 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.449571    2572 scope.go:117] "RemoveContainer" containerID="0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813"
Jul 20 11:21:27.449880 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:27.449859    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813\": container with ID starting with 0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813 not found: ID does not exist" containerID="0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813"
Jul 20 11:21:27.449930 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.449890    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813"} err="failed to get container status \"0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813\": rpc error: code = NotFound desc = could not find container \"0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813\": container with ID starting with 0ab18b737a34f00c47fb2bd100240b878c1c7d666b429cdf0376dfbfdb2b9813 not found: ID does not exist"
Jul 20 11:21:27.449930 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.449909    2572 scope.go:117] "RemoveContainer" containerID="835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8"
Jul 20 11:21:27.450128 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:21:27.450111    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8\": container with ID starting with 835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8 not found: ID does not exist" containerID="835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8"
Jul 20 11:21:27.450173 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.450132    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8"} err="failed to get container status \"835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8\": rpc error: code = NotFound desc = could not find container \"835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8\": container with ID starting with 835f34e9811c118da82a4e61fd2770e8bc70d89b16b1d519aacda56b28e85cc8 not found: ID does not exist"
Jul 20 11:21:27.554768 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.554676    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-home\") pod \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") "
Jul 20 11:21:27.554768 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.554720    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs\") pod \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") "
Jul 20 11:21:27.554768 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.554754    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkbqq\" (UniqueName: \"kubernetes.io/projected/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kube-api-access-jkbqq\") pod \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") "
Jul 20 11:21:27.555011 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.554783    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-model-cache\") pod \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") "
Jul 20 11:21:27.555011 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.554814    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tmp-dir\") pod \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") "
Jul 20 11:21:27.555011 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.554883    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-dshm\") pod \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") "
Jul 20 11:21:27.555011 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.554924    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kserve-provision-location\") pod \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\" (UID: \"041be8f8-2cf7-44e3-9b50-883dd8b26a7f\") "
Jul 20 11:21:27.555226 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.555078    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "041be8f8-2cf7-44e3-9b50-883dd8b26a7f" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:27.555466 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.555434    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-home" (OuterVolumeSpecName: "home") pod "041be8f8-2cf7-44e3-9b50-883dd8b26a7f" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:27.557013 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.556989    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kube-api-access-jkbqq" (OuterVolumeSpecName: "kube-api-access-jkbqq") pod "041be8f8-2cf7-44e3-9b50-883dd8b26a7f" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f"). InnerVolumeSpecName "kube-api-access-jkbqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:21:27.557121 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.557053    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-dshm" (OuterVolumeSpecName: "dshm") pod "041be8f8-2cf7-44e3-9b50-883dd8b26a7f" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:27.557317 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.557293    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "041be8f8-2cf7-44e3-9b50-883dd8b26a7f" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:21:27.568265 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.568233    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "041be8f8-2cf7-44e3-9b50-883dd8b26a7f" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:27.618368 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.618321    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "041be8f8-2cf7-44e3-9b50-883dd8b26a7f" (UID: "041be8f8-2cf7-44e3-9b50-883dd8b26a7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:21:27.656112 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.656089    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:27.656112 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.656114    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kserve-provision-location\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:27.656253 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.656123    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:27.656253 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.656134    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:27.656253 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.656147    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkbqq\" (UniqueName: \"kubernetes.io/projected/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-kube-api-access-jkbqq\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:27.656253 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.656156    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:27.656253 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.656165    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/041be8f8-2cf7-44e3-9b50-883dd8b26a7f-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:21:27.754512 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.754475    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 20 11:21:27.757818 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:27.757790    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 20 11:21:28.519547 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:28.519513    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" path="/var/lib/kubelet/pods/041be8f8-2cf7-44e3-9b50-883dd8b26a7f/volumes"
Jul 20 11:21:38.398194 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398147    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"]
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398619    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" containerName="storage-initializer"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398640    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" containerName="storage-initializer"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398657    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="storage-initializer"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398666    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="storage-initializer"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398674    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398683    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398699    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" containerName="main"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398705    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" containerName="main"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398761    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee80a3ca-a152-4f4b-b440-015d9f01150a" containerName="main"
Jul 20 11:21:38.400682 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.398771    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="041be8f8-2cf7-44e3-9b50-883dd8b26a7f" containerName="main"
Jul 20 11:21:38.401675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.401654    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.404679 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.404657    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 20 11:21:38.405810 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.405793    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 20 11:21:38.405886 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.405813    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 20 11:21:38.405973 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.405957    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-stfhh\""
Jul 20 11:21:38.410632 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.410602    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"]
Jul 20 11:21:38.433283 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.433255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.433439 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.433289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.433439 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.433324    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.433439 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.433363    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c344856-f88c-4ded-99df-8daebe06e0d1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.433546 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.433454    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.433546 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.433487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52qx\" (UniqueName: \"kubernetes.io/projected/9c344856-f88c-4ded-99df-8daebe06e0d1-kube-api-access-j52qx\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.534482 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.534446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.534482 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.534483    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c344856-f88c-4ded-99df-8daebe06e0d1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.534790 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.534743    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.534860 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.534796    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52qx\" (UniqueName: \"kubernetes.io/projected/9c344856-f88c-4ded-99df-8daebe06e0d1-kube-api-access-j52qx\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.534860 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.534840    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.534972 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.534883    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.535277 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.535251    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.535390 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.535290    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.535390 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.535379    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.536817 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.536796    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.537095 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.537078    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c344856-f88c-4ded-99df-8daebe06e0d1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.544429 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.544405    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52qx\" (UniqueName: \"kubernetes.io/projected/9c344856-f88c-4ded-99df-8daebe06e0d1-kube-api-access-j52qx\") pod \"scheduler-inline-config-test-kserve-8655bc757-tbl54\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.640361 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.640320    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"]
Jul 20 11:21:38.644239 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.644217    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.646875 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.646849    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-vhm9f\""
Jul 20 11:21:38.655471 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.655444    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"]
Jul 20 11:21:38.712345 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.712316    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:38.736498 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.736466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.736709 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.736519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2qqc\" (UniqueName: \"kubernetes.io/projected/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-kube-api-access-d2qqc\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.736709 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.736606    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.837395 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.837352    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.837595 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.837511    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2qqc\" (UniqueName: \"kubernetes.io/projected/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-kube-api-access-d2qqc\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.837664 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.837608    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.837978 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.837953    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.839924 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.839903    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.844861 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.844833    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"]
Jul 20 11:21:38.847145 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.847121    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2qqc\" (UniqueName: \"kubernetes.io/projected/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-kube-api-access-d2qqc\") pod \"scheduler-inline-config-test-kserve-router-scheduler-758579kpb2\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:38.849665 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:21:38.849642    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c344856_f88c_4ded_99df_8daebe06e0d1.slice/crio-21c457b8785d8d2fff8cbcb8dd0f03fa5ebf336ccfece677cee62971eafce926 WatchSource:0}: Error finding container 21c457b8785d8d2fff8cbcb8dd0f03fa5ebf336ccfece677cee62971eafce926: Status 404 returned error can't find the container with id 21c457b8785d8d2fff8cbcb8dd0f03fa5ebf336ccfece677cee62971eafce926
Jul 20 11:21:38.955348 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:38.955312    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:39.083653 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.083610    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"]
Jul 20 11:21:39.085728 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:21:39.085695    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82c30f0_8f25_40be_bf5b_780d35e7c2c7.slice/crio-d7daea342b2c0476d4681b34e744c6b91297741d08479c261dbe994e62d54530 WatchSource:0}: Error finding container d7daea342b2c0476d4681b34e744c6b91297741d08479c261dbe994e62d54530: Status 404 returned error can't find the container with id d7daea342b2c0476d4681b34e744c6b91297741d08479c261dbe994e62d54530
Jul 20 11:21:39.487740 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.487703    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54" event={"ID":"9c344856-f88c-4ded-99df-8daebe06e0d1","Type":"ContainerStarted","Data":"f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf"}
Jul 20 11:21:39.488191 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.487763    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54" event={"ID":"9c344856-f88c-4ded-99df-8daebe06e0d1","Type":"ContainerStarted","Data":"21c457b8785d8d2fff8cbcb8dd0f03fa5ebf336ccfece677cee62971eafce926"}
Jul 20 11:21:39.489188 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.489157    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2" event={"ID":"a82c30f0-8f25-40be-bf5b-780d35e7c2c7","Type":"ContainerStarted","Data":"a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506"}
Jul 20 11:21:39.489301 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.489204    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2" event={"ID":"a82c30f0-8f25-40be-bf5b-780d35e7c2c7","Type":"ContainerStarted","Data":"d7daea342b2c0476d4681b34e744c6b91297741d08479c261dbe994e62d54530"}
Jul 20 11:21:39.489301 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.489276    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:21:39.506652 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.506605    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54" podStartSLOduration=1.506591608 podStartE2EDuration="1.506591608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:21:39.505318728 +0000 UTC m=+5079.518420617" watchObservedRunningTime="2026-07-20 11:21:39.506591608 +0000 UTC m=+5079.519693503"
Jul 20 11:21:39.522421 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.522372    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2" podStartSLOduration=1.522358631 podStartE2EDuration="1.522358631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:21:39.521232076 +0000 UTC m=+5079.534333973" watchObservedRunningTime="2026-07-20 11:21:39.522358631 +0000 UTC m=+5079.535460527"
Jul 20 11:21:39.713216 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:39.713185    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:48.712899 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:48.712859    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:48.725464 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:48.725432    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:21:49.544985 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:21:49.544946    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:22:00.785227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:00.785133    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:22:00.823143 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:00.823114    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:22:10.495922 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:10.495885    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:22:12.685842 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:12.685807    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"]
Jul 20 11:22:12.703573 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:12.703526    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"]
Jul 20 11:22:12.711963 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:22:12.711922    2572 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 11:22:12.711963 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:22:12.711963 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:22:12.711963 ip-10-0-138-196 kubenswrapper[2572]:  > containerID="f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf" cmd=["/bin/sleep","15"]
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:12.711984    2572 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54" message="" err=<
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]:  >
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:22:12.712003    2572 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54" podUID="9c344856-f88c-4ded-99df-8daebe06e0d1" containerName="main" containerID="cri-o://f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf"
Jul 20 11:22:12.712502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:12.712046    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54" podUID="9c344856-f88c-4ded-99df-8daebe06e0d1" containerName="main" containerID="cri-o://f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf" gracePeriod=60
Jul 20 11:22:13.075030 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.075003    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:22:13.227342 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227282    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j52qx\" (UniqueName: \"kubernetes.io/projected/9c344856-f88c-4ded-99df-8daebe06e0d1-kube-api-access-j52qx\") pod \"9c344856-f88c-4ded-99df-8daebe06e0d1\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") "
Jul 20 11:22:13.227535 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227364    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-model-cache\") pod \"9c344856-f88c-4ded-99df-8daebe06e0d1\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") "
Jul 20 11:22:13.227535 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227395    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-dshm\") pod \"9c344856-f88c-4ded-99df-8daebe06e0d1\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") "
Jul 20 11:22:13.227535 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227428    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c344856-f88c-4ded-99df-8daebe06e0d1-tls-certs\") pod \"9c344856-f88c-4ded-99df-8daebe06e0d1\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") "
Jul 20 11:22:13.227535 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227463    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-tmp-dir\") pod \"9c344856-f88c-4ded-99df-8daebe06e0d1\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") "
Jul 20 11:22:13.227535 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227488    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-home\") pod \"9c344856-f88c-4ded-99df-8daebe06e0d1\" (UID: \"9c344856-f88c-4ded-99df-8daebe06e0d1\") "
Jul 20 11:22:13.227878 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227695    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c344856-f88c-4ded-99df-8daebe06e0d1" (UID: "9c344856-f88c-4ded-99df-8daebe06e0d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:22:13.227878 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227833    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c344856-f88c-4ded-99df-8daebe06e0d1" (UID: "9c344856-f88c-4ded-99df-8daebe06e0d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:22:13.227878 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.227861    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-home" (OuterVolumeSpecName: "home") pod "9c344856-f88c-4ded-99df-8daebe06e0d1" (UID: "9c344856-f88c-4ded-99df-8daebe06e0d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:22:13.229785 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.229759    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c344856-f88c-4ded-99df-8daebe06e0d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c344856-f88c-4ded-99df-8daebe06e0d1" (UID: "9c344856-f88c-4ded-99df-8daebe06e0d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:22:13.229785 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.229762    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c344856-f88c-4ded-99df-8daebe06e0d1-kube-api-access-j52qx" (OuterVolumeSpecName: "kube-api-access-j52qx") pod "9c344856-f88c-4ded-99df-8daebe06e0d1" (UID: "9c344856-f88c-4ded-99df-8daebe06e0d1"). InnerVolumeSpecName "kube-api-access-j52qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:22:13.229930 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.229808    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-dshm" (OuterVolumeSpecName: "dshm") pod "9c344856-f88c-4ded-99df-8daebe06e0d1" (UID: "9c344856-f88c-4ded-99df-8daebe06e0d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:22:13.328755 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.328704    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:13.328755 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.328753    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c344856-f88c-4ded-99df-8daebe06e0d1-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:13.328755 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.328765    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:13.329021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.328776    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:13.329021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.328789    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52qx\" (UniqueName: \"kubernetes.io/projected/9c344856-f88c-4ded-99df-8daebe06e0d1-kube-api-access-j52qx\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:13.329021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.328800    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c344856-f88c-4ded-99df-8daebe06e0d1-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:13.634437 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.634337    2572 generic.go:356] "Generic (PLEG): container finished" podID="9c344856-f88c-4ded-99df-8daebe06e0d1" containerID="f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf" exitCode=0
Jul 20 11:22:13.634437 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.634391    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"
Jul 20 11:22:13.634700 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.634430    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54" event={"ID":"9c344856-f88c-4ded-99df-8daebe06e0d1","Type":"ContainerDied","Data":"f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf"}
Jul 20 11:22:13.634700 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.634498    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54" event={"ID":"9c344856-f88c-4ded-99df-8daebe06e0d1","Type":"ContainerDied","Data":"21c457b8785d8d2fff8cbcb8dd0f03fa5ebf336ccfece677cee62971eafce926"}
Jul 20 11:22:13.634700 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.634531    2572 scope.go:117] "RemoveContainer" containerID="f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf"
Jul 20 11:22:13.644756 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.644733    2572 scope.go:117] "RemoveContainer" containerID="f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf"
Jul 20 11:22:13.645077 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:22:13.645053    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf\": container with ID starting with f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf not found: ID does not exist" containerID="f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf"
Jul 20 11:22:13.645133 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.645087    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf"} err="failed to get container status \"f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf\": rpc error: code = NotFound desc = could not find container \"f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf\": container with ID starting with f849f3fb7ddb05193a7d65aafe3805d86ad181c5b9890e312d724649c99725bf not found: ID does not exist"
Jul 20 11:22:13.655904 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.655877    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"]
Jul 20 11:22:13.660469 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:13.660438    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-tbl54"]
Jul 20 11:22:14.519762 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:14.519722    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c344856-f88c-4ded-99df-8daebe06e0d1" path="/var/lib/kubelet/pods/9c344856-f88c-4ded-99df-8daebe06e0d1/volumes"
Jul 20 11:22:20.495000 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:20.494955    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2" podUID="a82c30f0-8f25-40be-bf5b-780d35e7c2c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:22:27.706211 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:27.706119    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2" podUID="a82c30f0-8f25-40be-bf5b-780d35e7c2c7" containerName="main" containerID="cri-o://a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506" gracePeriod=45
Jul 20 11:22:27.957990 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:27.957921    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:22:28.051324 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.051282    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2qqc\" (UniqueName: \"kubernetes.io/projected/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-kube-api-access-d2qqc\") pod \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") "
Jul 20 11:22:28.051502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.051348    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tokenizer-uds\") pod \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") "
Jul 20 11:22:28.051502 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.051399    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tls-certs\") pod \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\" (UID: \"a82c30f0-8f25-40be-bf5b-780d35e7c2c7\") "
Jul 20 11:22:28.051712 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.051671    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a82c30f0-8f25-40be-bf5b-780d35e7c2c7" (UID: "a82c30f0-8f25-40be-bf5b-780d35e7c2c7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:22:28.053551 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.053528    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a82c30f0-8f25-40be-bf5b-780d35e7c2c7" (UID: "a82c30f0-8f25-40be-bf5b-780d35e7c2c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:22:28.053660 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.053586    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-kube-api-access-d2qqc" (OuterVolumeSpecName: "kube-api-access-d2qqc") pod "a82c30f0-8f25-40be-bf5b-780d35e7c2c7" (UID: "a82c30f0-8f25-40be-bf5b-780d35e7c2c7"). InnerVolumeSpecName "kube-api-access-d2qqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:22:28.153003 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.152961    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:28.153003 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.152990    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2qqc\" (UniqueName: \"kubernetes.io/projected/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-kube-api-access-d2qqc\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:28.153003 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.153002    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a82c30f0-8f25-40be-bf5b-780d35e7c2c7-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:22:28.692820 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.692782    2572 generic.go:356] "Generic (PLEG): container finished" podID="a82c30f0-8f25-40be-bf5b-780d35e7c2c7" containerID="a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506" exitCode=0
Jul 20 11:22:28.693027 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.692829    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"
Jul 20 11:22:28.693027 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.692854    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2" event={"ID":"a82c30f0-8f25-40be-bf5b-780d35e7c2c7","Type":"ContainerDied","Data":"a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506"}
Jul 20 11:22:28.693027 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.692909    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2" event={"ID":"a82c30f0-8f25-40be-bf5b-780d35e7c2c7","Type":"ContainerDied","Data":"d7daea342b2c0476d4681b34e744c6b91297741d08479c261dbe994e62d54530"}
Jul 20 11:22:28.693027 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.692931    2572 scope.go:117] "RemoveContainer" containerID="a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506"
Jul 20 11:22:28.701303 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.701279    2572 scope.go:117] "RemoveContainer" containerID="a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506"
Jul 20 11:22:28.701620 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:22:28.701599    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506\": container with ID starting with a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506 not found: ID does not exist" containerID="a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506"
Jul 20 11:22:28.701716 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.701629    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506"} err="failed to get container status \"a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506\": rpc error: code = NotFound desc = could not find container \"a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506\": container with ID starting with a6b5a94594526f1c0a3c7d3edda07bf53f0acf69d8013f519fedbc9ed0b67506 not found: ID does not exist"
Jul 20 11:22:28.709510 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.709474    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"]
Jul 20 11:22:28.712619 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:28.712595    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-758579kpb2"]
Jul 20 11:22:30.522733 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:30.522693    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82c30f0-8f25-40be-bf5b-780d35e7c2c7" path="/var/lib/kubelet/pods/a82c30f0-8f25-40be-bf5b-780d35e7c2c7/volumes"
Jul 20 11:22:52.287231 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.287179    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"]
Jul 20 11:22:52.287751 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.287688    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82c30f0-8f25-40be-bf5b-780d35e7c2c7" containerName="main"
Jul 20 11:22:52.287751 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.287710    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82c30f0-8f25-40be-bf5b-780d35e7c2c7" containerName="main"
Jul 20 11:22:52.287751 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.287739    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c344856-f88c-4ded-99df-8daebe06e0d1" containerName="main"
Jul 20 11:22:52.287751 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.287748    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c344856-f88c-4ded-99df-8daebe06e0d1" containerName="main"
Jul 20 11:22:52.287895 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.287833    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c344856-f88c-4ded-99df-8daebe06e0d1" containerName="main"
Jul 20 11:22:52.287895 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.287850    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82c30f0-8f25-40be-bf5b-780d35e7c2c7" containerName="main"
Jul 20 11:22:52.292379 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.292353    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.295947 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.295919    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 20 11:22:52.296422 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.295920    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-d9glq\""
Jul 20 11:22:52.296637 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.295941    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 20 11:22:52.296741 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.296015    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 20 11:22:52.296741 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.296505    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"]
Jul 20 11:22:52.333209 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.333172    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7drn\" (UniqueName: \"kubernetes.io/projected/d8d06904-e99e-4399-9813-6172398c4846-kube-api-access-q7drn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.333412 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.333255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d06904-e99e-4399-9813-6172398c4846-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.333412 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.333302    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.333412 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.333341    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.333412 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.333365    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.333412 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.333393    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.434631 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.434574    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d06904-e99e-4399-9813-6172398c4846-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.434631 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.434632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.434910 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.434652    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.434910 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.434680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.434910 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.434714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.434910 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.434771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7drn\" (UniqueName: \"kubernetes.io/projected/d8d06904-e99e-4399-9813-6172398c4846-kube-api-access-q7drn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.435129 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.435094    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.435129 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.435115    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.435211 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.435162    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.437040 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.437016    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.437315 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.437298    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d06904-e99e-4399-9813-6172398c4846-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.443427 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.443398    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7drn\" (UniqueName: \"kubernetes.io/projected/d8d06904-e99e-4399-9813-6172398c4846-kube-api-access-q7drn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.596433 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.596348    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"]
Jul 20 11:22:52.599713 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.599686    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.602357 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.602333    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-jmhsm\""
Jul 20 11:22:52.603966 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.603943    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:22:52.613105 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.613077    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"]
Jul 20 11:22:52.637010 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.636974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8311e4ed-621a-4b55-bcf0-8ce958f32681-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.637186 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.637091    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8sgp\" (UniqueName: \"kubernetes.io/projected/8311e4ed-621a-4b55-bcf0-8ce958f32681-kube-api-access-p8sgp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.637186 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.637129    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8311e4ed-621a-4b55-bcf0-8ce958f32681-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.738026 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.737818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8sgp\" (UniqueName: \"kubernetes.io/projected/8311e4ed-621a-4b55-bcf0-8ce958f32681-kube-api-access-p8sgp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.738026 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.737869    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8311e4ed-621a-4b55-bcf0-8ce958f32681-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.738026 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.737920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8311e4ed-621a-4b55-bcf0-8ce958f32681-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.738574 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.738528    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8311e4ed-621a-4b55-bcf0-8ce958f32681-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.739217 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.739194    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"]
Jul 20 11:22:52.740769 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.740747    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8311e4ed-621a-4b55-bcf0-8ce958f32681-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.742916 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:22:52.742890    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d06904_e99e_4399_9813_6172398c4846.slice/crio-551c388ce028f812171732ee0f1fb848e0669028afd87c6c7774fe368c0fd18c WatchSource:0}: Error finding container 551c388ce028f812171732ee0f1fb848e0669028afd87c6c7774fe368c0fd18c: Status 404 returned error can't find the container with id 551c388ce028f812171732ee0f1fb848e0669028afd87c6c7774fe368c0fd18c
Jul 20 11:22:52.746058 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.746034    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8sgp\" (UniqueName: \"kubernetes.io/projected/8311e4ed-621a-4b55-bcf0-8ce958f32681-kube-api-access-p8sgp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:52.791149 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.791115    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2" event={"ID":"d8d06904-e99e-4399-9813-6172398c4846","Type":"ContainerStarted","Data":"551c388ce028f812171732ee0f1fb848e0669028afd87c6c7774fe368c0fd18c"}
Jul 20 11:22:52.912232 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:52.912139    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:53.041412 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:53.041373    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"]
Jul 20 11:22:53.044664 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:22:53.044641    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8311e4ed_621a_4b55_bcf0_8ce958f32681.slice/crio-462434144ad38808a3fbb95f63e906938a3574b19c4ba5bafe09c712c53344e6 WatchSource:0}: Error finding container 462434144ad38808a3fbb95f63e906938a3574b19c4ba5bafe09c712c53344e6: Status 404 returned error can't find the container with id 462434144ad38808a3fbb95f63e906938a3574b19c4ba5bafe09c712c53344e6
Jul 20 11:22:53.797000 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:53.796952    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks" event={"ID":"8311e4ed-621a-4b55-bcf0-8ce958f32681","Type":"ContainerStarted","Data":"e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9"}
Jul 20 11:22:53.797544 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:53.797020    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks" event={"ID":"8311e4ed-621a-4b55-bcf0-8ce958f32681","Type":"ContainerStarted","Data":"462434144ad38808a3fbb95f63e906938a3574b19c4ba5bafe09c712c53344e6"}
Jul 20 11:22:53.797544 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:53.797136    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:22:53.798263 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:53.798239    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2" event={"ID":"d8d06904-e99e-4399-9813-6172398c4846","Type":"ContainerStarted","Data":"9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540"}
Jul 20 11:22:53.831805 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:53.831759    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks" podStartSLOduration=1.831744879 podStartE2EDuration="1.831744879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:22:53.831531708 +0000 UTC m=+5153.844633607" watchObservedRunningTime="2026-07-20 11:22:53.831744879 +0000 UTC m=+5153.844846776"
Jul 20 11:22:53.851716 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:53.851670    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2" podStartSLOduration=1.851654156 podStartE2EDuration="1.851654156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:22:53.850501085 +0000 UTC m=+5153.863603005" watchObservedRunningTime="2026-07-20 11:22:53.851654156 +0000 UTC m=+5153.864756054"
Jul 20 11:22:54.604610 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:22:54.604552    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:23:02.604383 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:02.604340    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:23:02.616930 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:02.616898    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:23:02.847447 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:02.847413    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:23:24.805015 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:24.804973    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:23:26.327153 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.327110    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"]
Jul 20 11:23:26.330089 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.330049    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"]
Jul 20 11:23:26.339030 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:23:26.338989    2572 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 11:23:26.339030 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:23:26.339030 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:23:26.339030 ip-10-0-138-196 kubenswrapper[2572]:  > containerID="9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540" cmd=["/bin/sleep","15"]
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.339054    2572 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2" message="" err=<
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]:  >
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:23:26.339070    2572 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2" podUID="d8d06904-e99e-4399-9813-6172398c4846" containerName="main" containerID="cri-o://9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540"
Jul 20 11:23:26.339343 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.339126    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2" podUID="d8d06904-e99e-4399-9813-6172398c4846" containerName="main" containerID="cri-o://9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540" gracePeriod=60
Jul 20 11:23:26.692093 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.692067    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:23:26.840869 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.840834    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d06904-e99e-4399-9813-6172398c4846-tls-certs\") pod \"d8d06904-e99e-4399-9813-6172398c4846\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") "
Jul 20 11:23:26.841062 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.840894    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7drn\" (UniqueName: \"kubernetes.io/projected/d8d06904-e99e-4399-9813-6172398c4846-kube-api-access-q7drn\") pod \"d8d06904-e99e-4399-9813-6172398c4846\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") "
Jul 20 11:23:26.841062 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.840933    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-model-cache\") pod \"d8d06904-e99e-4399-9813-6172398c4846\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") "
Jul 20 11:23:26.841062 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.840958    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-dshm\") pod \"d8d06904-e99e-4399-9813-6172398c4846\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") "
Jul 20 11:23:26.841062 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.840974    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-tmp-dir\") pod \"d8d06904-e99e-4399-9813-6172398c4846\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") "
Jul 20 11:23:26.841062 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.840996    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-home\") pod \"d8d06904-e99e-4399-9813-6172398c4846\" (UID: \"d8d06904-e99e-4399-9813-6172398c4846\") "
Jul 20 11:23:26.841376 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.841181    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8d06904-e99e-4399-9813-6172398c4846" (UID: "d8d06904-e99e-4399-9813-6172398c4846"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:23:26.841376 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.841305    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8d06904-e99e-4399-9813-6172398c4846" (UID: "d8d06904-e99e-4399-9813-6172398c4846"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:23:26.841376 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.841333    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-home" (OuterVolumeSpecName: "home") pod "d8d06904-e99e-4399-9813-6172398c4846" (UID: "d8d06904-e99e-4399-9813-6172398c4846"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:23:26.843311 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.843279    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-dshm" (OuterVolumeSpecName: "dshm") pod "d8d06904-e99e-4399-9813-6172398c4846" (UID: "d8d06904-e99e-4399-9813-6172398c4846"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:23:26.843424 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.843317    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d06904-e99e-4399-9813-6172398c4846-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8d06904-e99e-4399-9813-6172398c4846" (UID: "d8d06904-e99e-4399-9813-6172398c4846"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:23:26.843424 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.843372    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d06904-e99e-4399-9813-6172398c4846-kube-api-access-q7drn" (OuterVolumeSpecName: "kube-api-access-q7drn") pod "d8d06904-e99e-4399-9813-6172398c4846" (UID: "d8d06904-e99e-4399-9813-6172398c4846"). InnerVolumeSpecName "kube-api-access-q7drn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:23:26.930257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.930165    2572 generic.go:356] "Generic (PLEG): container finished" podID="d8d06904-e99e-4399-9813-6172398c4846" containerID="9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540" exitCode=0
Jul 20 11:23:26.930257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.930216    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"
Jul 20 11:23:26.930257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.930237    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2" event={"ID":"d8d06904-e99e-4399-9813-6172398c4846","Type":"ContainerDied","Data":"9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540"}
Jul 20 11:23:26.930499 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.930280    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2" event={"ID":"d8d06904-e99e-4399-9813-6172398c4846","Type":"ContainerDied","Data":"551c388ce028f812171732ee0f1fb848e0669028afd87c6c7774fe368c0fd18c"}
Jul 20 11:23:26.930499 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.930302    2572 scope.go:117] "RemoveContainer" containerID="9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540"
Jul 20 11:23:26.939318 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.939299    2572 scope.go:117] "RemoveContainer" containerID="9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540"
Jul 20 11:23:26.939625 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:23:26.939601    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540\": container with ID starting with 9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540 not found: ID does not exist" containerID="9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540"
Jul 20 11:23:26.939734 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.939633    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540"} err="failed to get container status \"9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540\": rpc error: code = NotFound desc = could not find container \"9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540\": container with ID starting with 9fc0012ddef8161adaabfb741904660e891771d3db3b91cb91a31da4dea64540 not found: ID does not exist"
Jul 20 11:23:26.941999 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.941982    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:26.942069 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.942002    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d06904-e99e-4399-9813-6172398c4846-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:26.942069 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.942013    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7drn\" (UniqueName: \"kubernetes.io/projected/d8d06904-e99e-4399-9813-6172398c4846-kube-api-access-q7drn\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:26.942069 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.942023    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:26.942069 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.942031    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:26.942069 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.942039    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d06904-e99e-4399-9813-6172398c4846-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:26.952003 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.951973    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"]
Jul 20 11:23:26.955917 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:26.955888    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f4bcd2"]
Jul 20 11:23:28.519253 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:28.519216    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d06904-e99e-4399-9813-6172398c4846" path="/var/lib/kubelet/pods/d8d06904-e99e-4399-9813-6172398c4846/volumes"
Jul 20 11:23:34.804308 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:34.804256    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks" podUID="8311e4ed-621a-4b55-bcf0-8ce958f32681" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:23:41.352239 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.352172    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks" podUID="8311e4ed-621a-4b55-bcf0-8ce958f32681" containerName="main" containerID="cri-o://e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9" gracePeriod=45
Jul 20 11:23:41.602361 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.602298    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:23:41.648757 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.648721    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8311e4ed-621a-4b55-bcf0-8ce958f32681-tokenizer-uds\") pod \"8311e4ed-621a-4b55-bcf0-8ce958f32681\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") "
Jul 20 11:23:41.648757 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.648762    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8311e4ed-621a-4b55-bcf0-8ce958f32681-tls-certs\") pod \"8311e4ed-621a-4b55-bcf0-8ce958f32681\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") "
Jul 20 11:23:41.649035 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.648839    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8sgp\" (UniqueName: \"kubernetes.io/projected/8311e4ed-621a-4b55-bcf0-8ce958f32681-kube-api-access-p8sgp\") pod \"8311e4ed-621a-4b55-bcf0-8ce958f32681\" (UID: \"8311e4ed-621a-4b55-bcf0-8ce958f32681\") "
Jul 20 11:23:41.649107 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.649027    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8311e4ed-621a-4b55-bcf0-8ce958f32681-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8311e4ed-621a-4b55-bcf0-8ce958f32681" (UID: "8311e4ed-621a-4b55-bcf0-8ce958f32681"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:23:41.650975 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.650951    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8311e4ed-621a-4b55-bcf0-8ce958f32681-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8311e4ed-621a-4b55-bcf0-8ce958f32681" (UID: "8311e4ed-621a-4b55-bcf0-8ce958f32681"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:23:41.651066 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.651046    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8311e4ed-621a-4b55-bcf0-8ce958f32681-kube-api-access-p8sgp" (OuterVolumeSpecName: "kube-api-access-p8sgp") pod "8311e4ed-621a-4b55-bcf0-8ce958f32681" (UID: "8311e4ed-621a-4b55-bcf0-8ce958f32681"). InnerVolumeSpecName "kube-api-access-p8sgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:23:41.749788 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.749757    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8sgp\" (UniqueName: \"kubernetes.io/projected/8311e4ed-621a-4b55-bcf0-8ce958f32681-kube-api-access-p8sgp\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:41.749788 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.749783    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8311e4ed-621a-4b55-bcf0-8ce958f32681-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:41.749788 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.749792    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8311e4ed-621a-4b55-bcf0-8ce958f32681-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:23:41.992311 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.992276    2572 generic.go:356] "Generic (PLEG): container finished" podID="8311e4ed-621a-4b55-bcf0-8ce958f32681" containerID="e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9" exitCode=0
Jul 20 11:23:41.992501 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.992324    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"
Jul 20 11:23:41.992501 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.992334    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks" event={"ID":"8311e4ed-621a-4b55-bcf0-8ce958f32681","Type":"ContainerDied","Data":"e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9"}
Jul 20 11:23:41.992501 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.992375    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks" event={"ID":"8311e4ed-621a-4b55-bcf0-8ce958f32681","Type":"ContainerDied","Data":"462434144ad38808a3fbb95f63e906938a3574b19c4ba5bafe09c712c53344e6"}
Jul 20 11:23:41.992501 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:41.992397    2572 scope.go:117] "RemoveContainer" containerID="e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9"
Jul 20 11:23:42.001124 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:42.001102    2572 scope.go:117] "RemoveContainer" containerID="e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9"
Jul 20 11:23:42.001407 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:23:42.001385    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9\": container with ID starting with e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9 not found: ID does not exist" containerID="e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9"
Jul 20 11:23:42.001485 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:42.001416    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9"} err="failed to get container status \"e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9\": rpc error: code = NotFound desc = could not find container \"e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9\": container with ID starting with e7a618fcd6b27b9d32f0fabe52d91c83b0a48d9b015fc425e6848b2d36bf06f9 not found: ID does not exist"
Jul 20 11:23:42.013247 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:42.013216    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"]
Jul 20 11:23:42.016790 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:42.016761    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheblrks"]
Jul 20 11:23:42.519567 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:42.519515    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8311e4ed-621a-4b55-bcf0-8ce958f32681" path="/var/lib/kubelet/pods/8311e4ed-621a-4b55-bcf0-8ce958f32681/volumes"
Jul 20 11:23:51.589270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.589238    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"]
Jul 20 11:23:51.589697 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.589526    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d06904-e99e-4399-9813-6172398c4846" containerName="main"
Jul 20 11:23:51.589697 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.589537    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d06904-e99e-4399-9813-6172398c4846" containerName="main"
Jul 20 11:23:51.589697 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.589548    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8311e4ed-621a-4b55-bcf0-8ce958f32681" containerName="main"
Jul 20 11:23:51.589697 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.589569    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8311e4ed-621a-4b55-bcf0-8ce958f32681" containerName="main"
Jul 20 11:23:51.589697 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.589619    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d06904-e99e-4399-9813-6172398c4846" containerName="main"
Jul 20 11:23:51.589697 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.589629    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8311e4ed-621a-4b55-bcf0-8ce958f32681" containerName="main"
Jul 20 11:23:51.593856 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.593834    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.597900 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.597870    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 20 11:23:51.598035 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.597901    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 20 11:23:51.598035 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.597901    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 20 11:23:51.598035 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.597870    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-r25nd\""
Jul 20 11:23:51.603121 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.603097    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"]
Jul 20 11:23:51.633598 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.633547    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb89r\" (UniqueName: \"kubernetes.io/projected/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-kube-api-access-cb89r\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.633773 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.633605    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.633773 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.633683    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.633773 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.633728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.633773 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.633760    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.633942 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.633783    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734306 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734511 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734333    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734511 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734374    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734572    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb89r\" (UniqueName: \"kubernetes.io/projected/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-kube-api-access-cb89r\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734604    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734838 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734691    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734838 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734759    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.734971 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.734912    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.736930 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.736900    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.737052 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.736929    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.744612 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.744589    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb89r\" (UniqueName: \"kubernetes.io/projected/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-kube-api-access-cb89r\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-66c78\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.872901 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.872815    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"]
Jul 20 11:23:51.876276 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.876256    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:51.878767 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.878742    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-bcwfs\""
Jul 20 11:23:51.884704 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.884675    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"]
Jul 20 11:23:51.904322 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.904287    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:23:51.936983 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.936944    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdsb\" (UniqueName: \"kubernetes.io/projected/d8b60b14-0c21-486c-a922-72cc7ea55a69-kube-api-access-fpdsb\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:51.936983 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.936984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8b60b14-0c21-486c-a922-72cc7ea55a69-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:51.937257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:51.937024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b60b14-0c21-486c-a922-72cc7ea55a69-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:52.037313 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.037285    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"]
Jul 20 11:23:52.037704 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.037669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdsb\" (UniqueName: \"kubernetes.io/projected/d8b60b14-0c21-486c-a922-72cc7ea55a69-kube-api-access-fpdsb\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:52.037768 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.037699    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8b60b14-0c21-486c-a922-72cc7ea55a69-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:52.037768 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.037744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b60b14-0c21-486c-a922-72cc7ea55a69-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:52.038091 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.038071    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8b60b14-0c21-486c-a922-72cc7ea55a69-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:52.039996 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:23:52.039969    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7e8bb2_bf2f_48a6_b62d_ddb660609f83.slice/crio-e9d38431f07b035b2a15f08d8f114f59127b50890ff8be367306af9ab1f5b8d9 WatchSource:0}: Error finding container e9d38431f07b035b2a15f08d8f114f59127b50890ff8be367306af9ab1f5b8d9: Status 404 returned error can't find the container with id e9d38431f07b035b2a15f08d8f114f59127b50890ff8be367306af9ab1f5b8d9
Jul 20 11:23:52.040292 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.040274    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b60b14-0c21-486c-a922-72cc7ea55a69-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:52.045786 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.045765    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdsb\" (UniqueName: \"kubernetes.io/projected/d8b60b14-0c21-486c-a922-72cc7ea55a69-kube-api-access-fpdsb\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:52.187190 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.187086    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:52.321260 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:52.321165    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"]
Jul 20 11:23:52.323819 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:23:52.323791    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b60b14_0c21_486c_a922_72cc7ea55a69.slice/crio-155695ff0a705c09debc643fc6c67ec26b4e291325816d93932aee7ceed9dd78 WatchSource:0}: Error finding container 155695ff0a705c09debc643fc6c67ec26b4e291325816d93932aee7ceed9dd78: Status 404 returned error can't find the container with id 155695ff0a705c09debc643fc6c67ec26b4e291325816d93932aee7ceed9dd78
Jul 20 11:23:53.040407 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:53.040370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s" event={"ID":"d8b60b14-0c21-486c-a922-72cc7ea55a69","Type":"ContainerStarted","Data":"1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb"}
Jul 20 11:23:53.040901 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:53.040422    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s" event={"ID":"d8b60b14-0c21-486c-a922-72cc7ea55a69","Type":"ContainerStarted","Data":"155695ff0a705c09debc643fc6c67ec26b4e291325816d93932aee7ceed9dd78"}
Jul 20 11:23:53.040901 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:53.040454    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:23:53.041949 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:53.041922    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78" event={"ID":"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83","Type":"ContainerStarted","Data":"69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927"}
Jul 20 11:23:53.042050 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:53.041968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78" event={"ID":"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83","Type":"ContainerStarted","Data":"e9d38431f07b035b2a15f08d8f114f59127b50890ff8be367306af9ab1f5b8d9"}
Jul 20 11:23:53.057246 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:53.057197    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s" podStartSLOduration=2.057182578 podStartE2EDuration="2.057182578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:23:53.0557557 +0000 UTC m=+5213.068857596" watchObservedRunningTime="2026-07-20 11:23:53.057182578 +0000 UTC m=+5213.070284503"
Jul 20 11:23:53.072319 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:53.072265    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78" podStartSLOduration=2.072251982 podStartE2EDuration="2.072251982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:23:53.070462559 +0000 UTC m=+5213.083564468" watchObservedRunningTime="2026-07-20 11:23:53.072251982 +0000 UTC m=+5213.085353877"
Jul 20 11:23:53.904687 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:23:53.904619    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:24:01.905135 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:01.905090    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:24:01.917807 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:01.917772    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:24:02.091245 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:02.091217    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:24:24.047689 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:24.047656    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:24:25.608162 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:25.608124    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"]
Jul 20 11:24:25.612163 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:25.612126    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"]
Jul 20 11:24:25.617897 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:24:25.617857    2572 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 11:24:25.617897 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:25.617897 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:25.617897 ip-10-0-138-196 kubenswrapper[2572]:  > containerID="69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927" cmd=["/bin/sleep","15"]
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:25.617917    2572 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78" message="" err=<
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]:  >
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:24:25.617932    2572 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78" podUID="5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" containerName="main" containerID="cri-o://69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927"
Jul 20 11:24:25.618200 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:25.617987    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78" podUID="5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" containerName="main" containerID="cri-o://69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927" gracePeriod=60
Jul 20 11:24:25.961166 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:25.961134    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:24:26.033494 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.033457    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tmp-dir\") pod \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") "
Jul 20 11:24:26.033715 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.033509    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-home\") pod \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") "
Jul 20 11:24:26.033715 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.033539    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tls-certs\") pod \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") "
Jul 20 11:24:26.033715 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.033588    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb89r\" (UniqueName: \"kubernetes.io/projected/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-kube-api-access-cb89r\") pod \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") "
Jul 20 11:24:26.033715 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.033615    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-dshm\") pod \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") "
Jul 20 11:24:26.033715 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.033637    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-model-cache\") pod \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\" (UID: \"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83\") "
Jul 20 11:24:26.033966 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.033826    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-home" (OuterVolumeSpecName: "home") pod "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" (UID: "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:26.033966 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.033839    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" (UID: "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:26.034074 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.034005    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" (UID: "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:26.034193 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.034174    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:26.034257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.034199    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:26.034257 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.034213    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:26.036057 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.036026    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-kube-api-access-cb89r" (OuterVolumeSpecName: "kube-api-access-cb89r") pod "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" (UID: "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83"). InnerVolumeSpecName "kube-api-access-cb89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:24:26.036057 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.036046    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-dshm" (OuterVolumeSpecName: "dshm") pod "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" (UID: "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:26.036216 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.036101    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" (UID: "5d7e8bb2-bf2f-48a6-b62d-ddb660609f83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:24:26.134908 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.134808    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb89r\" (UniqueName: \"kubernetes.io/projected/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-kube-api-access-cb89r\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:26.134908 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.134840    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:26.134908 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.134851    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:26.172624 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.172583    2572 generic.go:356] "Generic (PLEG): container finished" podID="5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" containerID="69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927" exitCode=0
Jul 20 11:24:26.172836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.172638    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"
Jul 20 11:24:26.172836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.172682    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78" event={"ID":"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83","Type":"ContainerDied","Data":"69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927"}
Jul 20 11:24:26.172836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.172751    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78" event={"ID":"5d7e8bb2-bf2f-48a6-b62d-ddb660609f83","Type":"ContainerDied","Data":"e9d38431f07b035b2a15f08d8f114f59127b50890ff8be367306af9ab1f5b8d9"}
Jul 20 11:24:26.172836 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.172781    2572 scope.go:117] "RemoveContainer" containerID="69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927"
Jul 20 11:24:26.181404 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.181388    2572 scope.go:117] "RemoveContainer" containerID="69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927"
Jul 20 11:24:26.181712 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:24:26.181690    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927\": container with ID starting with 69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927 not found: ID does not exist" containerID="69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927"
Jul 20 11:24:26.181764 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.181721    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927"} err="failed to get container status \"69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927\": rpc error: code = NotFound desc = could not find container \"69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927\": container with ID starting with 69d8c1e75fe54ee1e7e4985aaca284a56a98462046a3c5a89bcceeaa37779927 not found: ID does not exist"
Jul 20 11:24:26.193473 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.193445    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"]
Jul 20 11:24:26.196495 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.196471    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-66c78"]
Jul 20 11:24:26.520010 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:26.519975    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" path="/var/lib/kubelet/pods/5d7e8bb2-bf2f-48a6-b62d-ddb660609f83/volumes"
Jul 20 11:24:34.048511 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:34.048471    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s" podUID="d8b60b14-0c21-486c-a922-72cc7ea55a69" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:24:40.632462 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:40.632414    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s" podUID="d8b60b14-0c21-486c-a922-72cc7ea55a69" containerName="main" containerID="cri-o://1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb" gracePeriod=45
Jul 20 11:24:40.890678 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:40.890616    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:24:40.951315 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:40.951279    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8b60b14-0c21-486c-a922-72cc7ea55a69-tokenizer-uds\") pod \"d8b60b14-0c21-486c-a922-72cc7ea55a69\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") "
Jul 20 11:24:40.951315 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:40.951321    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b60b14-0c21-486c-a922-72cc7ea55a69-tls-certs\") pod \"d8b60b14-0c21-486c-a922-72cc7ea55a69\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") "
Jul 20 11:24:40.951592 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:40.951344    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpdsb\" (UniqueName: \"kubernetes.io/projected/d8b60b14-0c21-486c-a922-72cc7ea55a69-kube-api-access-fpdsb\") pod \"d8b60b14-0c21-486c-a922-72cc7ea55a69\" (UID: \"d8b60b14-0c21-486c-a922-72cc7ea55a69\") "
Jul 20 11:24:40.951672 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:40.951645    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b60b14-0c21-486c-a922-72cc7ea55a69-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d8b60b14-0c21-486c-a922-72cc7ea55a69" (UID: "d8b60b14-0c21-486c-a922-72cc7ea55a69"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:40.953533 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:40.953507    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b60b14-0c21-486c-a922-72cc7ea55a69-kube-api-access-fpdsb" (OuterVolumeSpecName: "kube-api-access-fpdsb") pod "d8b60b14-0c21-486c-a922-72cc7ea55a69" (UID: "d8b60b14-0c21-486c-a922-72cc7ea55a69"). InnerVolumeSpecName "kube-api-access-fpdsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:24:40.953603 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:40.953518    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b60b14-0c21-486c-a922-72cc7ea55a69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8b60b14-0c21-486c-a922-72cc7ea55a69" (UID: "d8b60b14-0c21-486c-a922-72cc7ea55a69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:24:41.052492 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.052448    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpdsb\" (UniqueName: \"kubernetes.io/projected/d8b60b14-0c21-486c-a922-72cc7ea55a69-kube-api-access-fpdsb\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:41.052492 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.052483    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8b60b14-0c21-486c-a922-72cc7ea55a69-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:41.052492 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.052493    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b60b14-0c21-486c-a922-72cc7ea55a69-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:24:41.234818 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.234782    2572 generic.go:356] "Generic (PLEG): container finished" podID="d8b60b14-0c21-486c-a922-72cc7ea55a69" containerID="1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb" exitCode=0
Jul 20 11:24:41.235013 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.234827    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"
Jul 20 11:24:41.235013 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.234844    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s" event={"ID":"d8b60b14-0c21-486c-a922-72cc7ea55a69","Type":"ContainerDied","Data":"1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb"}
Jul 20 11:24:41.235013 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.234888    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s" event={"ID":"d8b60b14-0c21-486c-a922-72cc7ea55a69","Type":"ContainerDied","Data":"155695ff0a705c09debc643fc6c67ec26b4e291325816d93932aee7ceed9dd78"}
Jul 20 11:24:41.235013 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.234909    2572 scope.go:117] "RemoveContainer" containerID="1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb"
Jul 20 11:24:41.251986 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.251961    2572 scope.go:117] "RemoveContainer" containerID="1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb"
Jul 20 11:24:41.252357 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:24:41.252333    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb\": container with ID starting with 1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb not found: ID does not exist" containerID="1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb"
Jul 20 11:24:41.252467 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.252361    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb"} err="failed to get container status \"1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb\": rpc error: code = NotFound desc = could not find container \"1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb\": container with ID starting with 1750443cb299894297e5a758be4b55ad7888b890a189d3d651899034530972cb not found: ID does not exist"
Jul 20 11:24:41.280641 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.280606    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"]
Jul 20 11:24:41.281837 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:41.281815    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-78857g8t7s"]
Jul 20 11:24:42.519442 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:42.519407    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b60b14-0c21-486c-a922-72cc7ea55a69" path="/var/lib/kubelet/pods/d8b60b14-0c21-486c-a922-72cc7ea55a69/volumes"
Jul 20 11:24:54.698624 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.698584    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"]
Jul 20 11:24:54.699021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.698884    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b60b14-0c21-486c-a922-72cc7ea55a69" containerName="main"
Jul 20 11:24:54.699021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.698895    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b60b14-0c21-486c-a922-72cc7ea55a69" containerName="main"
Jul 20 11:24:54.699021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.698906    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" containerName="main"
Jul 20 11:24:54.699021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.698912    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" containerName="main"
Jul 20 11:24:54.699021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.698955    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7e8bb2-bf2f-48a6-b62d-ddb660609f83" containerName="main"
Jul 20 11:24:54.699021 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.698963    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b60b14-0c21-486c-a922-72cc7ea55a69" containerName="main"
Jul 20 11:24:54.703269 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.703249    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.707072 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.707044    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 20 11:24:54.707227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.707076    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 20 11:24:54.707227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.707045    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-s7p25\""
Jul 20 11:24:54.707227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.707044    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 20 11:24:54.710052 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.710017    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"]
Jul 20 11:24:54.770419 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.770386    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36a71353-35d9-4da2-ad8c-07f6f4f63332-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.770419 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.770423    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.770712 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.770457    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.770712 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.770482    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.770712 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.770525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6cn\" (UniqueName: \"kubernetes.io/projected/36a71353-35d9-4da2-ad8c-07f6f4f63332-kube-api-access-kh6cn\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.770712 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.770572    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.871481 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.871438    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.871481 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.871480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.871746 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.871500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6cn\" (UniqueName: \"kubernetes.io/projected/36a71353-35d9-4da2-ad8c-07f6f4f63332-kube-api-access-kh6cn\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.871746 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.871532    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.871746 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.871589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36a71353-35d9-4da2-ad8c-07f6f4f63332-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.871746 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.871609    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.872069 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.872037    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.872069 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.872054    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.872220 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.872092    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.873942 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.873916    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.874098 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.874078    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36a71353-35d9-4da2-ad8c-07f6f4f63332-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:54.879003 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:54.878976    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6cn\" (UniqueName: \"kubernetes.io/projected/36a71353-35d9-4da2-ad8c-07f6f4f63332-kube-api-access-kh6cn\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:55.014075 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.014032    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:55.116896 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.116859    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"]
Jul 20 11:24:55.121040 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.121015    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.123935 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.123906    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-8pr62\""
Jul 20 11:24:55.135777 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.135747    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"]
Jul 20 11:24:55.157339 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.157312    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"]
Jul 20 11:24:55.160145 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:24:55.160112    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a71353_35d9_4da2_ad8c_07f6f4f63332.slice/crio-c73caa7cb94e92743d5cdfd81201d06f7eac0a41c932c737a6ec5a7d3fff9cd8 WatchSource:0}: Error finding container c73caa7cb94e92743d5cdfd81201d06f7eac0a41c932c737a6ec5a7d3fff9cd8: Status 404 returned error can't find the container with id c73caa7cb94e92743d5cdfd81201d06f7eac0a41c932c737a6ec5a7d3fff9cd8
Jul 20 11:24:55.174250 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.174219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0929121f-18b0-4573-a39c-1bf04fdfc108-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.174330 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.174289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79bmc\" (UniqueName: \"kubernetes.io/projected/0929121f-18b0-4573-a39c-1bf04fdfc108-kube-api-access-79bmc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.174376 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.174331    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0929121f-18b0-4573-a39c-1bf04fdfc108-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.275196 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.275099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0929121f-18b0-4573-a39c-1bf04fdfc108-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.275196 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.275165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bmc\" (UniqueName: \"kubernetes.io/projected/0929121f-18b0-4573-a39c-1bf04fdfc108-kube-api-access-79bmc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.275400 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.275200    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0929121f-18b0-4573-a39c-1bf04fdfc108-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.275694 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.275662    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0929121f-18b0-4573-a39c-1bf04fdfc108-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.277721 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.277701    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0929121f-18b0-4573-a39c-1bf04fdfc108-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.283347 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.283319    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bmc\" (UniqueName: \"kubernetes.io/projected/0929121f-18b0-4573-a39c-1bf04fdfc108-kube-api-access-79bmc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.292997 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.292970    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" event={"ID":"36a71353-35d9-4da2-ad8c-07f6f4f63332","Type":"ContainerStarted","Data":"871353dbc85676795f565082d1974a60c6ee2a5d1d68c91e603a911c15fcfb9d"}
Jul 20 11:24:55.293141 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.293021    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" event={"ID":"36a71353-35d9-4da2-ad8c-07f6f4f63332","Type":"ContainerStarted","Data":"c73caa7cb94e92743d5cdfd81201d06f7eac0a41c932c737a6ec5a7d3fff9cd8"}
Jul 20 11:24:55.310684 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.310636    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" podStartSLOduration=1.310619872 podStartE2EDuration="1.310619872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:24:55.310057912 +0000 UTC m=+5275.323159810" watchObservedRunningTime="2026-07-20 11:24:55.310619872 +0000 UTC m=+5275.323721768"
Jul 20 11:24:55.435476 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.435437    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:24:55.570611 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:55.570505    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"]
Jul 20 11:24:55.573326 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:24:55.573296    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0929121f_18b0_4573_a39c_1bf04fdfc108.slice/crio-266568da09b2858e409848bd61b623235fd0c226fd76f533f308a58ccf5a16c8 WatchSource:0}: Error finding container 266568da09b2858e409848bd61b623235fd0c226fd76f533f308a58ccf5a16c8: Status 404 returned error can't find the container with id 266568da09b2858e409848bd61b623235fd0c226fd76f533f308a58ccf5a16c8
Jul 20 11:24:56.014878 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:56.014827    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:24:56.299863 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:56.299759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4" event={"ID":"0929121f-18b0-4573-a39c-1bf04fdfc108","Type":"ContainerStarted","Data":"c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5"}
Jul 20 11:24:56.299863 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:56.299815    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4" event={"ID":"0929121f-18b0-4573-a39c-1bf04fdfc108","Type":"ContainerStarted","Data":"266568da09b2858e409848bd61b623235fd0c226fd76f533f308a58ccf5a16c8"}
Jul 20 11:24:56.322164 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:56.322108    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4" podStartSLOduration=1.322091646 podStartE2EDuration="1.322091646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:24:56.320464588 +0000 UTC m=+5276.333566484" watchObservedRunningTime="2026-07-20 11:24:56.322091646 +0000 UTC m=+5276.335193542"
Jul 20 11:24:57.304306 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:24:57.304264    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:25:05.014713 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:05.014664    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:25:05.027212 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:05.027176    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:25:05.349090 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:05.348995    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:25:28.035533 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.035440    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"]
Jul 20 11:25:28.046015 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:25:28.045969    2572 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 11:25:28.046015 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:25:28.046015 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:25:28.046015 ip-10-0-138-196 kubenswrapper[2572]:  > containerID="871353dbc85676795f565082d1974a60c6ee2a5d1d68c91e603a911c15fcfb9d" cmd=["/bin/sleep","15"]
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.046037    2572 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" message="" err=<
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]:  >
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:25:28.046054    2572 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]:         , stdout: , stderr: , exit code -1
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" podUID="36a71353-35d9-4da2-ad8c-07f6f4f63332" containerName="main" containerID="cri-o://871353dbc85676795f565082d1974a60c6ee2a5d1d68c91e603a911c15fcfb9d"
Jul 20 11:25:28.046597 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.046098    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" podUID="36a71353-35d9-4da2-ad8c-07f6f4f63332" containerName="main" containerID="cri-o://871353dbc85676795f565082d1974a60c6ee2a5d1d68c91e603a911c15fcfb9d" gracePeriod=60
Jul 20 11:25:28.047640 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.047617    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"]
Jul 20 11:25:28.049403 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.049367    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4" podUID="0929121f-18b0-4573-a39c-1bf04fdfc108" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:25:28.338675 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.338574    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" podUID="36a71353-35d9-4da2-ad8c-07f6f4f63332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": dial tcp 10.134.0.90:8000: connect: connection refused"
Jul 20 11:25:28.434942 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.434901    2572 generic.go:356] "Generic (PLEG): container finished" podID="36a71353-35d9-4da2-ad8c-07f6f4f63332" containerID="871353dbc85676795f565082d1974a60c6ee2a5d1d68c91e603a911c15fcfb9d" exitCode=0
Jul 20 11:25:28.435118 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.434975    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" event={"ID":"36a71353-35d9-4da2-ad8c-07f6f4f63332","Type":"ContainerDied","Data":"871353dbc85676795f565082d1974a60c6ee2a5d1d68c91e603a911c15fcfb9d"}
Jul 20 11:25:28.791030 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.791002    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:25:28.852962 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.852926    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-dshm\") pod \"36a71353-35d9-4da2-ad8c-07f6f4f63332\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") "
Jul 20 11:25:28.852962 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.852972    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-home\") pod \"36a71353-35d9-4da2-ad8c-07f6f4f63332\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") "
Jul 20 11:25:28.853256 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.853024    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-model-cache\") pod \"36a71353-35d9-4da2-ad8c-07f6f4f63332\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") "
Jul 20 11:25:28.853256 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.853059    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh6cn\" (UniqueName: \"kubernetes.io/projected/36a71353-35d9-4da2-ad8c-07f6f4f63332-kube-api-access-kh6cn\") pod \"36a71353-35d9-4da2-ad8c-07f6f4f63332\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") "
Jul 20 11:25:28.853256 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.853084    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-tmp-dir\") pod \"36a71353-35d9-4da2-ad8c-07f6f4f63332\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") "
Jul 20 11:25:28.853256 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.853123    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36a71353-35d9-4da2-ad8c-07f6f4f63332-tls-certs\") pod \"36a71353-35d9-4da2-ad8c-07f6f4f63332\" (UID: \"36a71353-35d9-4da2-ad8c-07f6f4f63332\") "
Jul 20 11:25:28.853466 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.853328    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-home" (OuterVolumeSpecName: "home") pod "36a71353-35d9-4da2-ad8c-07f6f4f63332" (UID: "36a71353-35d9-4da2-ad8c-07f6f4f63332"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:25:28.853466 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.853370    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36a71353-35d9-4da2-ad8c-07f6f4f63332" (UID: "36a71353-35d9-4da2-ad8c-07f6f4f63332"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:25:28.853466 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.853384    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-model-cache" (OuterVolumeSpecName: "model-cache") pod "36a71353-35d9-4da2-ad8c-07f6f4f63332" (UID: "36a71353-35d9-4da2-ad8c-07f6f4f63332"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:25:28.855404 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.855382    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-dshm" (OuterVolumeSpecName: "dshm") pod "36a71353-35d9-4da2-ad8c-07f6f4f63332" (UID: "36a71353-35d9-4da2-ad8c-07f6f4f63332"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:25:28.855404 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.855389    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a71353-35d9-4da2-ad8c-07f6f4f63332-kube-api-access-kh6cn" (OuterVolumeSpecName: "kube-api-access-kh6cn") pod "36a71353-35d9-4da2-ad8c-07f6f4f63332" (UID: "36a71353-35d9-4da2-ad8c-07f6f4f63332"). InnerVolumeSpecName "kube-api-access-kh6cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:25:28.855532 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.855417    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a71353-35d9-4da2-ad8c-07f6f4f63332-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36a71353-35d9-4da2-ad8c-07f6f4f63332" (UID: "36a71353-35d9-4da2-ad8c-07f6f4f63332"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:25:28.954042 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.954005    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-model-cache\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:28.954042 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.954043    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6cn\" (UniqueName: \"kubernetes.io/projected/36a71353-35d9-4da2-ad8c-07f6f4f63332-kube-api-access-kh6cn\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:28.954240 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.954054    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-tmp-dir\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:28.954240 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.954064    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36a71353-35d9-4da2-ad8c-07f6f4f63332-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:28.954240 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.954073    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-dshm\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:28.954240 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:28.954080    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36a71353-35d9-4da2-ad8c-07f6f4f63332-home\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:29.440827 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:29.440731    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2" event={"ID":"36a71353-35d9-4da2-ad8c-07f6f4f63332","Type":"ContainerDied","Data":"c73caa7cb94e92743d5cdfd81201d06f7eac0a41c932c737a6ec5a7d3fff9cd8"}
Jul 20 11:25:29.440827 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:29.440761    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"
Jul 20 11:25:29.440827 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:29.440790    2572 scope.go:117] "RemoveContainer" containerID="871353dbc85676795f565082d1974a60c6ee2a5d1d68c91e603a911c15fcfb9d"
Jul 20 11:25:29.461710 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:29.461682    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"]
Jul 20 11:25:29.464778 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:29.464754    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-w4nj2"]
Jul 20 11:25:30.521130 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:30.521096    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a71353-35d9-4da2-ad8c-07f6f4f63332" path="/var/lib/kubelet/pods/36a71353-35d9-4da2-ad8c-07f6f4f63332/volumes"
Jul 20 11:25:38.049917 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:38.049874    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4" podUID="0929121f-18b0-4573-a39c-1bf04fdfc108" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 11:25:43.068317 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.068244    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4" podUID="0929121f-18b0-4573-a39c-1bf04fdfc108" containerName="main" containerID="cri-o://c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5" gracePeriod=45
Jul 20 11:25:43.311857 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.311833    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:25:43.368262 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.368183    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0929121f-18b0-4573-a39c-1bf04fdfc108-tls-certs\") pod \"0929121f-18b0-4573-a39c-1bf04fdfc108\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") "
Jul 20 11:25:43.368262 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.368226    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0929121f-18b0-4573-a39c-1bf04fdfc108-tokenizer-uds\") pod \"0929121f-18b0-4573-a39c-1bf04fdfc108\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") "
Jul 20 11:25:43.368477 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.368288    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79bmc\" (UniqueName: \"kubernetes.io/projected/0929121f-18b0-4573-a39c-1bf04fdfc108-kube-api-access-79bmc\") pod \"0929121f-18b0-4573-a39c-1bf04fdfc108\" (UID: \"0929121f-18b0-4573-a39c-1bf04fdfc108\") "
Jul 20 11:25:43.368589 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.368547    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0929121f-18b0-4573-a39c-1bf04fdfc108-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0929121f-18b0-4573-a39c-1bf04fdfc108" (UID: "0929121f-18b0-4573-a39c-1bf04fdfc108"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:25:43.370409 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.370386    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0929121f-18b0-4573-a39c-1bf04fdfc108-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0929121f-18b0-4573-a39c-1bf04fdfc108" (UID: "0929121f-18b0-4573-a39c-1bf04fdfc108"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:25:43.370525 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.370416    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0929121f-18b0-4573-a39c-1bf04fdfc108-kube-api-access-79bmc" (OuterVolumeSpecName: "kube-api-access-79bmc") pod "0929121f-18b0-4573-a39c-1bf04fdfc108" (UID: "0929121f-18b0-4573-a39c-1bf04fdfc108"). InnerVolumeSpecName "kube-api-access-79bmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:25:43.469259 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.469221    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0929121f-18b0-4573-a39c-1bf04fdfc108-tls-certs\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:43.469259 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.469251    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0929121f-18b0-4573-a39c-1bf04fdfc108-tokenizer-uds\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:43.469259 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.469260    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79bmc\" (UniqueName: \"kubernetes.io/projected/0929121f-18b0-4573-a39c-1bf04fdfc108-kube-api-access-79bmc\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:25:43.497674 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.497638    2572 generic.go:356] "Generic (PLEG): container finished" podID="0929121f-18b0-4573-a39c-1bf04fdfc108" containerID="c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5" exitCode=0
Jul 20 11:25:43.497853 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.497686    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"
Jul 20 11:25:43.497853 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.497707    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4" event={"ID":"0929121f-18b0-4573-a39c-1bf04fdfc108","Type":"ContainerDied","Data":"c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5"}
Jul 20 11:25:43.497853 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.497751    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4" event={"ID":"0929121f-18b0-4573-a39c-1bf04fdfc108","Type":"ContainerDied","Data":"266568da09b2858e409848bd61b623235fd0c226fd76f533f308a58ccf5a16c8"}
Jul 20 11:25:43.497853 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.497773    2572 scope.go:117] "RemoveContainer" containerID="c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5"
Jul 20 11:25:43.506684 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.506663    2572 scope.go:117] "RemoveContainer" containerID="c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5"
Jul 20 11:25:43.506932 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:25:43.506911    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5\": container with ID starting with c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5 not found: ID does not exist" containerID="c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5"
Jul 20 11:25:43.506999 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.506939    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5"} err="failed to get container status \"c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5\": rpc error: code = NotFound desc = could not find container \"c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5\": container with ID starting with c5e4f85e31473195690349a9099515f167f48f29d698c8cf6fed12799135a4c5 not found: ID does not exist"
Jul 20 11:25:43.518208 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.518183    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"]
Jul 20 11:25:43.521715 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:43.521694    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-68d9968q5tz4"]
Jul 20 11:25:44.519646 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:44.519612    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0929121f-18b0-4573-a39c-1bf04fdfc108" path="/var/lib/kubelet/pods/0929121f-18b0-4573-a39c-1bf04fdfc108/volumes"
Jul 20 11:25:49.445198 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.440403    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dwbhh/must-gather-sjl6j"]
Jul 20 11:25:49.445969 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.445942    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0929121f-18b0-4573-a39c-1bf04fdfc108" containerName="main"
Jul 20 11:25:49.445969 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.445971    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0929121f-18b0-4573-a39c-1bf04fdfc108" containerName="main"
Jul 20 11:25:49.446157 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.445995    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a71353-35d9-4da2-ad8c-07f6f4f63332" containerName="main"
Jul 20 11:25:49.446157 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.446003    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a71353-35d9-4da2-ad8c-07f6f4f63332" containerName="main"
Jul 20 11:25:49.446157 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.446085    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0929121f-18b0-4573-a39c-1bf04fdfc108" containerName="main"
Jul 20 11:25:49.446157 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.446100    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a71353-35d9-4da2-ad8c-07f6f4f63332" containerName="main"
Jul 20 11:25:49.450790 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.450763    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwbhh/must-gather-sjl6j"]
Jul 20 11:25:49.450917 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.450866    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:25:49.453722 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.453700    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwbhh\"/\"kube-root-ca.crt\""
Jul 20 11:25:49.454862 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.454843    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwbhh\"/\"openshift-service-ca.crt\""
Jul 20 11:25:49.454996 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.454886    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dwbhh\"/\"default-dockercfg-mv8sm\""
Jul 20 11:25:49.519634 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.519595    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-must-gather-output\") pod \"must-gather-sjl6j\" (UID: \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\") " pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:25:49.519829 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.519648    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x6lc\" (UniqueName: \"kubernetes.io/projected/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-kube-api-access-7x6lc\") pod \"must-gather-sjl6j\" (UID: \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\") " pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:25:49.620876 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.620833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-must-gather-output\") pod \"must-gather-sjl6j\" (UID: \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\") " pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:25:49.621057 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.620904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x6lc\" (UniqueName: \"kubernetes.io/projected/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-kube-api-access-7x6lc\") pod \"must-gather-sjl6j\" (UID: \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\") " pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:25:49.621199 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.621177    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-must-gather-output\") pod \"must-gather-sjl6j\" (UID: \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\") " pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:25:49.629754 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.629725    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x6lc\" (UniqueName: \"kubernetes.io/projected/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-kube-api-access-7x6lc\") pod \"must-gather-sjl6j\" (UID: \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\") " pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:25:49.760177 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.760130    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:25:49.889926 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.889892    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwbhh/must-gather-sjl6j"]
Jul 20 11:25:49.893006 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:25:49.892979    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b38ba3b_8c7a_4b2e_8506_177d4c0eecd5.slice/crio-60a53e1ac1136277247e38ef7a81c1f31ccf84998f67d5e29dabd946a523409f WatchSource:0}: Error finding container 60a53e1ac1136277247e38ef7a81c1f31ccf84998f67d5e29dabd946a523409f: Status 404 returned error can't find the container with id 60a53e1ac1136277247e38ef7a81c1f31ccf84998f67d5e29dabd946a523409f
Jul 20 11:25:49.894882 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:49.894864    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 11:25:50.524669 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:50.524634    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwbhh/must-gather-sjl6j" event={"ID":"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5","Type":"ContainerStarted","Data":"60a53e1ac1136277247e38ef7a81c1f31ccf84998f67d5e29dabd946a523409f"}
Jul 20 11:25:56.559479 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:56.559439    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwbhh/must-gather-sjl6j" event={"ID":"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5","Type":"ContainerStarted","Data":"da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc"}
Jul 20 11:25:56.559479 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:56.559487    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwbhh/must-gather-sjl6j" event={"ID":"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5","Type":"ContainerStarted","Data":"d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7"}
Jul 20 11:25:56.576170 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:25:56.576120    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dwbhh/must-gather-sjl6j" podStartSLOduration=1.347457292 podStartE2EDuration="7.576103206s" totalImagesPullingTime="6.228645914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:25:56.574509346 +0000 UTC m=+5336.587611243" watchObservedRunningTime="2026-07-20 11:25:56.576103206 +0000 UTC m=+5336.589205102"
Jul 20 11:26:14.225055 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:14.225020    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7fxgf_4820f7c1-e06a-403e-bbd4-d3b9d3ef0660/kuadrant-console-plugin/0.log"
Jul 20 11:26:19.660304 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:19.660269    2572 generic.go:356] "Generic (PLEG): container finished" podID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerID="d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7" exitCode=0
Jul 20 11:26:19.660791 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:19.660347    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwbhh/must-gather-sjl6j" event={"ID":"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5","Type":"ContainerDied","Data":"d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7"}
Jul 20 11:26:19.660791 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:19.660706    2572 scope.go:117] "RemoveContainer" containerID="d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7"
Jul 20 11:26:19.863644 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:19.863600    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dwbhh_must-gather-sjl6j_1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5/gather/0.log"
Jul 20 11:26:23.498894 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:23.498840    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n46hj_3700312c-9a42-4a43-a2e8-a7827283c09b/global-pull-secret-syncer/0.log"
Jul 20 11:26:23.688272 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:23.688236    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hchgr_2c7e6e5b-2a9d-48fd-bfb7-c7d6c4b5647b/konnectivity-agent/0.log"
Jul 20 11:26:23.760650 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:23.760552    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-196.ec2.internal_9e1c3e21b15be9fdfd60693e86ae0598/haproxy/0.log"
Jul 20 11:26:25.284227 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.284185    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-dwbhh/must-gather-sjl6j"]
Jul 20 11:26:25.284685 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.284377    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-dwbhh/must-gather-sjl6j" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerName="copy" containerID="cri-o://da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc" gracePeriod=2
Jul 20 11:26:25.286598 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:26:25.286542    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sjl6j\" is forbidden: User \"system:node:ip-10-0-138-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwbhh\": no relationship found between node 'ip-10-0-138-196.ec2.internal' and this object" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:26:25.286777 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.286760    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-dwbhh/must-gather-sjl6j"]
Jul 20 11:26:25.520441 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.520410    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dwbhh_must-gather-sjl6j_1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5/copy/0.log"
Jul 20 11:26:25.522794 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.522773    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:26:25.524862 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:26:25.524838    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sjl6j\" is forbidden: User \"system:node:ip-10-0-138-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwbhh\": no relationship found between node 'ip-10-0-138-196.ec2.internal' and this object" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:26:25.528110 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.528091    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-must-gather-output\") pod \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\" (UID: \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\") "
Jul 20 11:26:25.528161 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.528127    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x6lc\" (UniqueName: \"kubernetes.io/projected/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-kube-api-access-7x6lc\") pod \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\" (UID: \"1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5\") "
Jul 20 11:26:25.530343 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.530320    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-kube-api-access-7x6lc" (OuterVolumeSpecName: "kube-api-access-7x6lc") pod "1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" (UID: "1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5"). InnerVolumeSpecName "kube-api-access-7x6lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:26:25.533768 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.533742    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" (UID: "1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:26:25.628859 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.628806    2572 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-must-gather-output\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:26:25.628859 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.628852    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x6lc\" (UniqueName: \"kubernetes.io/projected/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5-kube-api-access-7x6lc\") on node \"ip-10-0-138-196.ec2.internal\" DevicePath \"\""
Jul 20 11:26:25.686650 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.686623    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dwbhh_must-gather-sjl6j_1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5/copy/0.log"
Jul 20 11:26:25.686996 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.686974    2572 generic.go:356] "Generic (PLEG): container finished" podID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerID="da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc" exitCode=143
Jul 20 11:26:25.687061 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.687027    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:26:25.687101 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.687064    2572 scope.go:117] "RemoveContainer" containerID="da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc"
Jul 20 11:26:25.689439 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:26:25.689415    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sjl6j\" is forbidden: User \"system:node:ip-10-0-138-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwbhh\": no relationship found between node 'ip-10-0-138-196.ec2.internal' and this object" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:26:25.694987 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.694967    2572 scope.go:117] "RemoveContainer" containerID="d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7"
Jul 20 11:26:25.697385 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:26:25.697360    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sjl6j\" is forbidden: User \"system:node:ip-10-0-138-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dwbhh\": no relationship found between node 'ip-10-0-138-196.ec2.internal' and this object" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" pod="openshift-must-gather-dwbhh/must-gather-sjl6j"
Jul 20 11:26:25.708215 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.708195    2572 scope.go:117] "RemoveContainer" containerID="da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc"
Jul 20 11:26:25.708517 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:26:25.708492    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc\": container with ID starting with da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc not found: ID does not exist" containerID="da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc"
Jul 20 11:26:25.708652 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.708526    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc"} err="failed to get container status \"da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc\": rpc error: code = NotFound desc = could not find container \"da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc\": container with ID starting with da15ff66759d8fd2c17740392f49f6aeb2870fdb4747346287149f453131c4cc not found: ID does not exist"
Jul 20 11:26:25.708652 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.708544    2572 scope.go:117] "RemoveContainer" containerID="d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7"
Jul 20 11:26:25.708846 ip-10-0-138-196 kubenswrapper[2572]: E0720 11:26:25.708822    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7\": container with ID starting with d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7 not found: ID does not exist" containerID="d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7"
Jul 20 11:26:25.708889 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:25.708854    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7"} err="failed to get container status \"d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7\": rpc error: code = NotFound desc = could not find container \"d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7\": container with ID starting with d9fe06b1e3c4188908b426da31c1b4c7b37272e2aef9bf0dd4d29ef779c5d1b7 not found: ID does not exist"
Jul 20 11:26:26.525938 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:26.525901    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" path="/var/lib/kubelet/pods/1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5/volumes"
Jul 20 11:26:28.389945 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:28.389917    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7fxgf_4820f7c1-e06a-403e-bbd4-d3b9d3ef0660/kuadrant-console-plugin/0.log"
Jul 20 11:26:30.574462 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:30.574431    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6m2w_7ec3ea3d-aff0-4370-b384-5c59e9c08ba6/node-exporter/0.log"
Jul 20 11:26:30.596987 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:30.596957    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6m2w_7ec3ea3d-aff0-4370-b384-5c59e9c08ba6/kube-rbac-proxy/0.log"
Jul 20 11:26:30.625161 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:30.625127    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6m2w_7ec3ea3d-aff0-4370-b384-5c59e9c08ba6/init-textfile/0.log"
Jul 20 11:26:30.976941 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:30.976908    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9jx8b_c93d2f2a-5a06-4bea-a170-db5aaa435782/prometheus-operator/0.log"
Jul 20 11:26:31.001137 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:31.001106    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9jx8b_c93d2f2a-5a06-4bea-a170-db5aaa435782/kube-rbac-proxy/0.log"
Jul 20 11:26:31.030605 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:31.030578    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-5kt4c_b15c8705-6e49-4c81-a40d-5d867e84f9c3/prometheus-operator-admission-webhook/0.log"
Jul 20 11:26:32.675118 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:32.675086    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-6zs8d_59b0e12e-76ad-4a85-b34a-46f765f89fa8/networking-console-plugin/0.log"
Jul 20 11:26:33.713698 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:33.713663    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-8n9sc_cebf0a31-f95c-4ea3-baf6-7e4b2e3fcf7a/download-server/0.log"
Jul 20 11:26:34.122270 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.122183    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"]
Jul 20 11:26:34.122477 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.122465    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerName="copy"
Jul 20 11:26:34.122522 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.122477    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerName="copy"
Jul 20 11:26:34.122522 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.122488    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerName="gather"
Jul 20 11:26:34.122522 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.122495    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerName="gather"
Jul 20 11:26:34.122667 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.122546    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerName="gather"
Jul 20 11:26:34.122667 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.122570    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b38ba3b-8c7a-4b2e-8506-177d4c0eecd5" containerName="copy"
Jul 20 11:26:34.124626 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.124609    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.127159 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.127132    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4rtcj\"/\"kube-root-ca.crt\""
Jul 20 11:26:34.127282 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.127198    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4rtcj\"/\"openshift-service-ca.crt\""
Jul 20 11:26:34.128439 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.128424    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4rtcj\"/\"default-dockercfg-dh72t\""
Jul 20 11:26:34.131815 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.131662    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"]
Jul 20 11:26:34.183179 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.183144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-podres\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.183179 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.183181    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-lib-modules\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.183403 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.183201    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-proc\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.183403 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.183218    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qmf\" (UniqueName: \"kubernetes.io/projected/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-kube-api-access-n9qmf\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.183403 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.183275    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-sys\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.283961 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.283919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-podres\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.283961 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.283967    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-lib-modules\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.284182 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.283990    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-proc\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.284182 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.284009    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qmf\" (UniqueName: \"kubernetes.io/projected/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-kube-api-access-n9qmf\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.284182 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.284030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-sys\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.284182 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.284102    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-podres\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.284182 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.284104    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-proc\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.284182 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.284139    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-lib-modules\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.284182 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.284111    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-sys\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.291720 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.291689    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qmf\" (UniqueName: \"kubernetes.io/projected/1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e-kube-api-access-n9qmf\") pod \"perf-node-gather-daemonset-g87wz\" (UID: \"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.436008 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.435909    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.562797 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.562761    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"]
Jul 20 11:26:34.566178 ip-10-0-138-196 kubenswrapper[2572]: W0720 11:26:34.566153    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1af0c3a4_e2e2_4ae8_adf4_278ed1f7e58e.slice/crio-07389bdfd6a502a8bc697b52f56acb5119913a11cdd7e3728e592b4a6952dbdb WatchSource:0}: Error finding container 07389bdfd6a502a8bc697b52f56acb5119913a11cdd7e3728e592b4a6952dbdb: Status 404 returned error can't find the container with id 07389bdfd6a502a8bc697b52f56acb5119913a11cdd7e3728e592b4a6952dbdb
Jul 20 11:26:34.723899 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.723864    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz" event={"ID":"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e","Type":"ContainerStarted","Data":"9e3847f37e92d7cc65b23582e02f2f57ecb82d8a27ba4c1529b4eb313eee7237"}
Jul 20 11:26:34.724336 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.723926    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz" event={"ID":"1af0c3a4-e2e2-4ae8-adf4-278ed1f7e58e","Type":"ContainerStarted","Data":"07389bdfd6a502a8bc697b52f56acb5119913a11cdd7e3728e592b4a6952dbdb"}
Jul 20 11:26:34.724336 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.724047    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:34.739429 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:34.739381    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz" podStartSLOduration=0.739365002 podStartE2EDuration="739.365002ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 11:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:26:34.737761496 +0000 UTC m=+5374.750863402" watchObservedRunningTime="2026-07-20 11:26:34.739365002 +0000 UTC m=+5374.752466898"
Jul 20 11:26:35.575267 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:35.575215    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s686d_a80acdcd-5608-4ba8-bc25-82c606e1751c/dns/0.log"
Jul 20 11:26:35.596957 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:35.596928    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s686d_a80acdcd-5608-4ba8-bc25-82c606e1751c/kube-rbac-proxy/0.log"
Jul 20 11:26:35.621974 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:35.621947    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2dxpj_0991b138-7f5a-47b5-9fa9-436a20cc6b6c/dns-node-resolver/0.log"
Jul 20 11:26:36.216597 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:36.216518    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c56464c7b-b9pvp_6d129d25-6c55-47d1-8fa1-d83bdd5540ba/registry/0.log"
Jul 20 11:26:36.247458 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:36.247426    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-66rz6_c443edf8-fe49-4b79-ae03-a78c32a1b5af/node-ca/0.log"
Jul 20 11:26:37.914068 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:37.914030    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kglbm_1dbf57eb-aeb4-4414-bd2d-2c263fed7126/serve-healthcheck-canary/0.log"
Jul 20 11:26:38.504022 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:38.503978    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fqs2h_21a853c5-75ff-4f78-8c9d-ad305c4d1a1d/kube-rbac-proxy/0.log"
Jul 20 11:26:38.525758 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:38.525724    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fqs2h_21a853c5-75ff-4f78-8c9d-ad305c4d1a1d/exporter/0.log"
Jul 20 11:26:38.547724 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:38.547688    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fqs2h_21a853c5-75ff-4f78-8c9d-ad305c4d1a1d/extractor/0.log"
Jul 20 11:26:40.736597 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:40.736570    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-g87wz"
Jul 20 11:26:41.777817 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:41.777788    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86dd6ccfdd-9mv22_36b3e193-fea9-4ce0-ae12-fe593edc284b/manager/0.log"
Jul 20 11:26:42.413876 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:42.413847    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-645f4f9567-g46rm_4516489e-95a6-4df3-8a02-4f7deac66e56/manager/0.log"
Jul 20 11:26:42.568301 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:42.568271    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-687957b4c5-s2t9c_c826efa8-2802-4184-933f-f8a98556eed8/manager/0.log"
Jul 20 11:26:43.239047 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:43.239016    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2bmzn_2dc82dae-2654-417e-b3a5-bcb4a46ff195/s3-init/0.log"
Jul 20 11:26:43.256260 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:43.256234    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2bmzn_2dc82dae-2654-417e-b3a5-bcb4a46ff195/download-hf-model/0.log"
Jul 20 11:26:49.745735 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:49.745700    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zp58_72ae4625-f4be-43b5-ab3e-2b10725394da/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:26:49.766835 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:49.766805    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zp58_72ae4625-f4be-43b5-ab3e-2b10725394da/egress-router-binary-copy/0.log"
Jul 20 11:26:49.788315 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:49.788286    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zp58_72ae4625-f4be-43b5-ab3e-2b10725394da/cni-plugins/0.log"
Jul 20 11:26:49.809907 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:49.809872    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zp58_72ae4625-f4be-43b5-ab3e-2b10725394da/bond-cni-plugin/0.log"
Jul 20 11:26:49.833088 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:49.833061    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zp58_72ae4625-f4be-43b5-ab3e-2b10725394da/routeoverride-cni/0.log"
Jul 20 11:26:49.854092 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:49.854069    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zp58_72ae4625-f4be-43b5-ab3e-2b10725394da/whereabouts-cni-bincopy/0.log"
Jul 20 11:26:49.875933 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:49.875906    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zp58_72ae4625-f4be-43b5-ab3e-2b10725394da/whereabouts-cni/0.log"
Jul 20 11:26:50.265782 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:50.265747    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lrbz6_a0c1a30d-7d22-4667-ac0c-eba67d79c668/kube-multus/0.log"
Jul 20 11:26:50.358293 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:50.358261    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-82lt6_f29da995-4bb0-47b2-9bd5-943406c9c977/network-metrics-daemon/0.log"
Jul 20 11:26:50.375828 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:50.375805    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-82lt6_f29da995-4bb0-47b2-9bd5-943406c9c977/kube-rbac-proxy/0.log"
Jul 20 11:26:51.579519 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.579482    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-controller/0.log"
Jul 20 11:26:51.608755 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.608724    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/1.log"
Jul 20 11:26:51.608926 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.608728    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovn-acl-logging/0.log"
Jul 20 11:26:51.627407 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.627382    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/kube-rbac-proxy-node/0.log"
Jul 20 11:26:51.648576 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.648543    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/kube-rbac-proxy-ovn-metrics/0.log"
Jul 20 11:26:51.668111 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.668084    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/northd/0.log"
Jul 20 11:26:51.688857 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.688833    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/nbdb/0.log"
Jul 20 11:26:51.709654 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.709621    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/sbdb/0.log"
Jul 20 11:26:51.818319 ip-10-0-138-196 kubenswrapper[2572]: I0720 11:26:51.818286    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4csm_f4785433-f502-4748-a0d9-c718625cdd56/ovnkube-controller/0.log"